Commit Graph

122 Commits

Author SHA1 Message Date
Kaz Wesley
c525b201b9
Parser: don't panic for any standard library files (#3609) 2022-07-28 19:17:33 +02:00
Kaz Wesley
c670718e3c
JNI bindings for enso-parser (#3599)
Provide a JNI dynamic-library interface to `enso_parser`.

# Important Notes
- The library can be built with: `cargo build -p enso-parser-jni`.
- A new `org.enso.syntax2.Parser` API is implemented on top of the JNI interface provided by `enso-parser-jni`.
- We are using the `jni` crate, since apparently Java cannot just call C-ABI functions. The crate is not well-maintained. I came across an obviously-unsound `safe` function, and found it was reported over a year ago, with a PR to fix: jni-rs/jni-rs#303. However our needs are simple. We can't trust any safety guarantees they imply, but I think we are unlikely to encounter any logic bugs using the basic bindings.
2022-07-25 14:24:21 +00:00
Kaz Wesley
3b99e18f94
Code blocks (#3585) 2022-07-20 16:53:20 +02:00
Kaz Wesley
100eeda673
Simple assignments and function definitions (#3572)
implement simple variable assignments and function definitions.

This implements:
- https://www.pivotaltracker.com/story/show/182497122
- https://www.pivotaltracker.com/story/show/182497144 (the code blocks are not created yet, but the function declaration is recognized.)

# Important Notes
- Introduced S-expression-based tests, and pretty-printing-roundtrip testing.
- Started writing tests for TypeDef based on the examples in the issue. None of them parse successfully.
- Fixed Number tokenizing.
- Moved most contents of parser's `main.rs` to `lib.rs` (fixes a warning).
2022-07-07 22:31:00 +00:00
Kaz Wesley
bc66078251
Parser: Transpile Rust AST types to Java types (#3555)
Implement generation of Java AST types from the Rust AST type definitions, with support for deserializing in Java syntax trees created in Rust.

### New Libraries

#### `enso-reflect`

Implements a `#[derive(Reflect)]` macro to enable runtime analysis of datatypes. Macro interface includes helper attributes; **the Rust types and the `reflect` attributes applied to them fully determine the Java types** ultimately produced (by `enso-metamodel`). This is the most important API, as it is used in the subject crates (`enso-parser`, and dependencies with types used in the AST). [Module docs](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/reflect/macros/src/lib.rs).

#### `enso-metamodel`

Provides data models for data models in Rust/Java/Meta (a highly-abstracted language-independent model--I have referred to it before as the "generic representation", but that was an overloaded term).

The high-level interface consists of operations on data models, and between them. For example, the only operations needed by [the binary that drives datatype transpilation](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/parser/generate-java/src/main.rs) are: `rust::to_meta`, `java::from_meta`, `java::transform::optional_to_null`, `java::to_syntax`.

The low-level interface consists of direct usage of the datatypes; this is used by [the module that implements some serialization overrides](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/parser/generate-java/src/serialization.rs) (so that the Java interface to `Code` references can produce `String`s on demand based on serialized offset/length pairs). The serialization override mechanism is based on customizing, not replacing, the generated deserialization methods, so as to be as robust as possible to changes in the Rust source or in the transpilation process.

### Important Notes

- Rust/Java serialization is exhaustively tested for structural compatibility. A function [`metamodel::meta::serialization::testcases`](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/metamodel/src/meta/serialization.rs) uses `reflect`-derived data to generate serialized representations of ASTs to use as test cases. Its should-accept cases cover every type a tree can contain; it also produces a representative set of should-reject cases. A Rust `#[test]` confirms that these cases are accepted/rejected as expected, and generated Java tests (see Binaries below) check the generated Java deserialization code against the same test cases.
- Deserializing `Code` is untested. The mechanism is in place (in Rust, we serialize only the offset/length of the `Cow`; in Java, during deserialization we obtain a context object holding a buffer for all string data; the accessor generated in Java uses the buffer and the offset/length to return `String`s), but it will be easier to test once we have implemented actually parsing something and instantiating the `Cow`s with source code.
- `#[tagged_enum]` [now supports](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/shapely/macros/src/tagged_enum.rs#L36-L51) control over what is done with container-level attributes; they can be applied to the container and variants (default), only to the container, or only to variants.
- Generation of `sealed` classes is supported, but currently disabled by `TARGET_VERSION` in `metamodel::java::syntax` so that tests don't require Java 15 to run. (The same logic is run either way; there is a shallow difference in output.)

### Binaries

The `enso-parser-generate-java` crate defines several binaries:
- `enso-parser-generate-java`: Performs the transpilation; after integration, this will be invoked by the build script.
- `java-tests`: Generates the Java code that tests format deserialization; after integration this command will be invoked by the build script, and its Java output compiled and run during testing.
- `graph-rust`/`graph-meta`/`graph-java`: Produce GraphViz representations of data models in different typesystems; these are for developing and understanding model transformations. 

Until integration, a **script regenerates the Java and runs the format tests: `./tools/parser_generate_java.sh`**. The generated code can be browsed in `target/generated_java`.
2022-07-07 04:46:42 +02:00
Wojciech Daniło
7c0330290f
Parser improvements (#3549) 2022-07-01 05:42:29 +02:00
Michał Wawrzyniec Urbańczyk
14a01c4635
New IDE build script (#3466) 2022-05-23 04:16:04 +02:00
Wojciech Daniło
9e219d698c
Initial parser implementation in Rust (#3341) 2022-05-17 05:13:20 +02:00
Wojciech Danilo
55dce7af65 Fixing build. 2022-03-10 06:21:57 +01:00
Wojciech Danilo
2accb722a8 Applying review 2022-03-10 05:57:59 +01:00
Wojciech Danilo
73f33d1c7f Linting codebase 2022-03-10 05:32:33 +01:00
Michael Mauderer
32cfb0333a
Bump Rust toolchain to nightly-2022-01-20. (#3255) 2022-02-16 12:58:02 +00:00
Adam Obuchowicz
c68ac5c0d6
Integration Test Framework (#3257) 2022-02-11 13:19:02 +01:00
Adam Obuchowicz
c4d22102cf
Switch to 2021 edition (#3173) 2021-12-01 16:06:57 +01:00
Adam Obuchowicz
9ab4f45e72
Refactored enso-data crate and text utilities. (#3166) 2021-11-25 11:45:42 +01:00
Adam Obuchowicz
753f787a6f
Refactored config crate + moving crates to lib/rust directory. (#3155) 2021-11-12 13:56:23 +01:00
Adam Obuchowicz
942464cbaf
Refactoring: merge utils into prelude; merge workspaces. (#3151) 2021-11-10 14:36:08 +01:00
Michał Wawrzyniec Urbańczyk
848cef7dca
Use libraries merged from rust-lib directly (#3137) 2021-11-05 16:55:55 +01:00
Ara Adkins
48bcebc723
Update to GraalVM 21.1.0 (#1738) 2021-05-14 13:08:39 +01:00
Dmitry Bushev
e2b583c64a
Return Correct Name for the Module in AtomConstructor (#1375)
return the module name if it's the atom constructor of the module.
2021-01-11 14:29:50 +03:00
Ara Adkins
4cc36e8c81
Preparation for more parser work (#1363) 2020-12-18 14:25:30 +00:00
Josef
5345bdcb3e
Stub Out the Parser Interface (#1065) 2020-08-14 11:10:52 +02:00