LadybirdBrowser/ladybird
1
1
Fork 0
mirror of https://github.com/LadybirdBrowser/ladybird.git synced 2024-10-04 17:17:32 +03:00
Code Issues Packages Projects Releases Wiki Activity
4e546cee97
ladybird/Userland/Libraries/LibUnicode/unicode_data.cmake

95 lines
5.1 KiB
CMake
Raw Normal View History

LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
option(ENABLE_UNICODE_DATABASE_DOWNLOAD "Enable download of Unicode UCD files at build time" ON)
set(UNICODE_DATA_URL https://www.unicode.org/Public/13.0.0/ucd/UnicodeData.txt)
set(UNICODE_DATA_PATH ${CMAKE_BINARY_DIR}/UCD/UnicodeData.txt)
LibUnicode: Download and parse the special casing UCD file This adds a SpecialCasing structure to the generated UnicodeData.h/cpp files. This structure contains casing rules for code points which have non-1-to-1 upper-to-lower case code point mappings. Further, these rules may be limited to specific locales or other context.
2021-07-27 17:39:37 +03:00
set(SPECIAL_CASING_URL https://www.unicode.org/Public/13.0.0/ucd/SpecialCasing.txt)
set(SPECIAL_CASING_PATH ${CMAKE_BINARY_DIR}/UCD/SpecialCasing.txt)
LibUnicode: Download and parse the property list UCD file
2021-07-28 00:37:47 +03:00
set(PROP_LIST_URL https://www.unicode.org/Public/13.0.0/ucd/PropList.txt)
set(PROP_LIST_PATH ${CMAKE_BINARY_DIR}/UCD/PropList.txt)
LibUnicode: Parse and utilize DerivedCoreProperties DerivedCoreProperties are pseudo-properties that are the union of other categories and properties. For example, the derived property Math is the union of the general category Sm and the property Other_Math. Parsing these is necessary for implementing Unicode property escapes. But it also has the added benefit that LibUnicode now does not need to derive some of these properties at runtime.
2021-07-29 01:39:41 +03:00
set(DERIVED_CORE_PROP_URL https://www.unicode.org/Public/13.0.0/ucd/DerivedCoreProperties.txt)
set(DERIVED_CORE_PROP_PATH ${CMAKE_BINARY_DIR}/UCD/DerivedCoreProperties.txt)
LibUnicode: Parse UCD DerivedBinaryProperties.txt and generate property
2021-08-04 14:53:24 +03:00
set(DERIVED_BINARY_PROP_URL https://www.unicode.org/Public/13.0.0/ucd/extracted/DerivedBinaryProperties.txt)
set(DERIVED_BINARY_PROP_PATH ${CMAKE_BINARY_DIR}/UCD/DerivedBinaryProperties.txt)
LibUnicode: Parse and generate PropertyAliases These are all used for Unicode property escapes.
2021-07-29 03:14:42 +03:00
set(PROP_ALIAS_URL https://www.unicode.org/Public/13.0.0/ucd/PropertyAliases.txt)
set(PROP_ALIAS_PATH ${CMAKE_BINARY_DIR}/UCD/PropertyAliases.txt)
LibUnicode: Generate General Category unions and aliases This downloads the PropertyValueAliases.txt UCD file, which contains a set of General Category aliases. This changes the General Category enumeration to now be generated as a bitmask. This is to easily allow General Category unions. For example, the LC (Cased_Letter) category is the union of the Ll, Lu, and Lt categories.
2021-07-31 20:45:05 +03:00
set(PROP_VALUE_ALIAS_URL https://www.unicode.org/Public/13.0.0/ucd/PropertyValueAliases.txt)
set(PROP_VALUE_ALIAS_PATH ${CMAKE_BINARY_DIR}/UCD/PropertyValueAliases.txt)
LibUnicode: Parse UCD Scripts.txt and generate as a Unicode property There are a couple of minor nuances with parsing script values, compared to other properties. In Scripts.txt, the UCD file lists the full name of each script; other properties, like General Category, list the shorter name in their primary files. This means that the aliases listed in PropertyValueAliases.txt are reversed for script values.
2021-08-04 00:11:19 +03:00
set(SCRIPTS_URL https://www.unicode.org/Public/13.0.0/ucd/Scripts.txt)
set(SCRIPTS_PATH ${CMAKE_BINARY_DIR}/UCD/Scripts.txt)
LibUnicode: Parse UCD ScriptExtensions.txt and generate property
2021-08-04 14:05:30 +03:00
set(SCRIPT_EXTENSIONS_URL https://www.unicode.org/Public/13.0.0/ucd/ScriptExtensions.txt)
set(SCRIPT_EXTENSIONS_PATH ${CMAKE_BINARY_DIR}/UCD/ScriptExtensions.txt)
LibUnicode: Parse UCD emoji-data.txt and generate Unicode property
2021-08-04 14:46:36 +03:00
set(EMOJI_DATA_URL https://www.unicode.org/Public/13.0.0/ucd/emoji/emoji-data.txt)
set(EMOJI_DATA_PATH ${CMAKE_BINARY_DIR}/UCD/emoji-data.txt)
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
if (ENABLE_UNICODE_DATABASE_DOWNLOAD)
if (NOT EXISTS ${UNICODE_DATA_PATH})
message(STATUS "Downloading UCD UnicodeData.txt from ${UNICODE_DATA_URL}...")
file(DOWNLOAD ${UNICODE_DATA_URL} ${UNICODE_DATA_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Download and parse the special casing UCD file This adds a SpecialCasing structure to the generated UnicodeData.h/cpp files. This structure contains casing rules for code points which have non-1-to-1 upper-to-lower case code point mappings. Further, these rules may be limited to specific locales or other context.
2021-07-27 17:39:37 +03:00
if (NOT EXISTS ${SPECIAL_CASING_PATH})
message(STATUS "Downloading UCD SpecialCasing.txt from ${SPECIAL_CASING_URL}...")
file(DOWNLOAD ${SPECIAL_CASING_URL} ${SPECIAL_CASING_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Download and parse the property list UCD file
2021-07-28 00:37:47 +03:00
if (NOT EXISTS ${PROP_LIST_PATH})
message(STATUS "Downloading UCD PropList.txt from ${PROP_LIST_URL}...")
file(DOWNLOAD ${PROP_LIST_URL} ${PROP_LIST_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Parse and utilize DerivedCoreProperties DerivedCoreProperties are pseudo-properties that are the union of other categories and properties. For example, the derived property Math is the union of the general category Sm and the property Other_Math. Parsing these is necessary for implementing Unicode property escapes. But it also has the added benefit that LibUnicode now does not need to derive some of these properties at runtime.
2021-07-29 01:39:41 +03:00
if (NOT EXISTS ${DERIVED_CORE_PROP_PATH})
message(STATUS "Downloading UCD DerivedCoreProperties.txt from ${DERIVED_CORE_PROP_URL}...")
file(DOWNLOAD ${DERIVED_CORE_PROP_URL} ${DERIVED_CORE_PROP_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Parse UCD DerivedBinaryProperties.txt and generate property
2021-08-04 14:53:24 +03:00
if (NOT EXISTS ${DERIVED_BINARY_PROP_PATH})
message(STATUS "Downloading UCD DerivedBinaryProperties.txt from ${DERIVED_BINARY_PROP_URL}...")
file(DOWNLOAD ${DERIVED_BINARY_PROP_URL} ${DERIVED_BINARY_PROP_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Parse and generate PropertyAliases These are all used for Unicode property escapes.
2021-07-29 03:14:42 +03:00
if (NOT EXISTS ${PROP_ALIAS_PATH})
message(STATUS "Downloading UCD PropertyAliases.txt from ${PROP_ALIAS_URL}...")
file(DOWNLOAD ${PROP_ALIAS_URL} ${PROP_ALIAS_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Generate General Category unions and aliases This downloads the PropertyValueAliases.txt UCD file, which contains a set of General Category aliases. This changes the General Category enumeration to now be generated as a bitmask. This is to easily allow General Category unions. For example, the LC (Cased_Letter) category is the union of the Ll, Lu, and Lt categories.
2021-07-31 20:45:05 +03:00
if (NOT EXISTS ${PROP_VALUE_ALIAS_PATH})
message(STATUS "Downloading UCD PropertyValueAliases.txt from ${PROP_VALUE_ALIAS_URL}...")
file(DOWNLOAD ${PROP_VALUE_ALIAS_URL} ${PROP_VALUE_ALIAS_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Parse UCD Scripts.txt and generate as a Unicode property There are a couple of minor nuances with parsing script values, compared to other properties. In Scripts.txt, the UCD file lists the full name of each script; other properties, like General Category, list the shorter name in their primary files. This means that the aliases listed in PropertyValueAliases.txt are reversed for script values.
2021-08-04 00:11:19 +03:00
if (NOT EXISTS ${SCRIPTS_PATH})
message(STATUS "Downloading UCD Scripts.txt from ${SCRIPTS_URL}...")
file(DOWNLOAD ${SCRIPTS_URL} ${SCRIPTS_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Parse UCD ScriptExtensions.txt and generate property
2021-08-04 14:05:30 +03:00
if (NOT EXISTS ${SCRIPT_EXTENSIONS_PATH})
message(STATUS "Downloading UCD ScriptExtensions.txt from ${SCRIPT_EXTENSIONS_URL}...")
file(DOWNLOAD ${SCRIPT_EXTENSIONS_URL} ${SCRIPT_EXTENSIONS_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Parse UCD emoji-data.txt and generate Unicode property
2021-08-04 14:46:36 +03:00
if (NOT EXISTS ${EMOJI_DATA_PATH})
message(STATUS "Downloading UCD emoji-data.txt from ${EMOJI_DATA_URL}...")
file(DOWNLOAD ${EMOJI_DATA_URL} ${EMOJI_DATA_PATH} INACTIVITY_TIMEOUT 10)
endif()
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
LibUnicode: Make unicode data generation logic more relocatable The previous logic had several checks for Lagom directories and subdirectories. All we really want to do for these header checks is make sure that the files end up in an included folder prefixed with LibUnicode. We also don't need to hard code the path to the generator, the $<TARGET_FILES> generator expression can create the path for us.
2021-07-29 11:18:52 +03:00
set(UNICODE_DATA_HEADER LibUnicode/UnicodeData.h)
set(UNICODE_DATA_IMPLEMENTATION LibUnicode/UnicodeData.cpp)
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
LibUnicode: Make unicode data generation logic more relocatable The previous logic had several checks for Lagom directories and subdirectories. All we really want to do for these header checks is make sure that the files end up in an included folder prefixed with LibUnicode. We also don't need to hard code the path to the generator, the $<TARGET_FILES> generator expression can create the path for us.
2021-07-29 11:18:52 +03:00
if (CMAKE_CURRENT_BINARY_DIR MATCHES ".*/LibUnicode") # Serenity build.
set(UNICODE_DATA_HEADER UnicodeData.h)
set(UNICODE_DATA_IMPLEMENTATION UnicodeData.cpp)
Meta: Add LibUnicode (and its tests) to Lagom This is primarily to allow using LibUnicode within LibJS and its REPL. Note: this seems to be the first time that a Lagom dependency requires generated source files. For this to work, some of Lagom's CMakeLists.txt commands needed to be re-organized to include the CMake files that fetch and parse UnicodeData.txt. The paths required to invoke the generator also differ depending on what is currently building (SerenityOS vs. Lagom as part of the Serenity build vs. a standalone Lagom build).
2021-07-25 22:15:47 +03:00
endif()
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
add_custom_command(
LibUnicode: Invoke Unicode data generator a single time It takes a non-neglible amount of time to parse all of the UCD files and generate the Unicode data files. To help compile times, only invoke the generator once.
2021-08-02 17:47:41 +03:00
OUTPUT ${UNICODE_DATA_HEADER} ${UNICODE_DATA_IMPLEMENTATION}
LibUnicode: Remove WordBreakProperty from generated Unicode data This was originally used for the "is_final_code_point" algorithm in LibUnicode/CharacterTypes.cpp. However, it has since been superseded by DerivedCoreProperties and is now unused. Remove it as it is currently a waste of time to process the data, and is trivial to add back if we need it again.
2021-08-10 02:03:01 +03:00
COMMAND $<TARGET_FILE:GenerateUnicodeData> -h ${UNICODE_DATA_HEADER} -c ${UNICODE_DATA_IMPLEMENTATION} -u ${UNICODE_DATA_PATH} -s ${SPECIAL_CASING_PATH} -p ${PROP_LIST_PATH} -d ${DERIVED_CORE_PROP_PATH} -b ${DERIVED_BINARY_PROP_PATH} -a ${PROP_ALIAS_PATH} -v ${PROP_VALUE_ALIAS_PATH} -r ${SCRIPTS_PATH} -x ${SCRIPT_EXTENSIONS_PATH} -e ${EMOJI_DATA_PATH}
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
VERBATIM
LibUnicode: Remove WordBreakProperty from generated Unicode data This was originally used for the "is_final_code_point" algorithm in LibUnicode/CharacterTypes.cpp. However, it has since been superseded by DerivedCoreProperties and is now unused. Remove it as it is currently a waste of time to process the data, and is trivial to add back if we need it again.
2021-08-10 02:03:01 +03:00
DEPENDS GenerateUnicodeData ${UNICODE_DATA_PATH} ${SPECIAL_CASING_PATH} ${PROP_LIST_PATH} ${DERIVED_CORE_PROP_PATH} ${DERIVED_BINARY_PROP_PATH} ${PROP_ALIAS_PATH} ${PROP_VALUE_ALIAS_PATH} ${SCRIPTS_PATH} ${SCRIPT_EXTENSIONS_PATH} ${EMOJI_DATA_PATH}
LibUnicode: Introduce a Unicode library for interacting with UCD files The Unicode standard publishes the Unicode Character Database (UCD) with information about every code point, such as each code point's upper case mapping. LibUnicode exists to download and parse UCD files at build time and to provide accessors to that data. As a start, LibUnicode includes upper- and lower-case code point converters.
2021-07-25 22:10:51 +03:00
)
set(UNICODE_DATA_SOURCES ${UNICODE_DATA_HEADER} ${UNICODE_DATA_IMPLEMENTATION})
add_compile_definitions(ENABLE_UNICODE_DATA=1)
else()
add_compile_definitions(ENABLE_UNICODE_DATA=0)
endif()
Reference in New Issue Copy Permalink