[ENHANCEMENT] Add TextMate (TM4E) syntax highlighting for generated text artifact file types in the CIMTool product

[tviegut]

Is your feature request related to a problem? Please describe:

CIMTool generates a wide range of textual artifacts through its XSLT builders — among them JSON, YAML, Java, C#, SQL, Python, and C++ source and schema files. When these artifacts are opened within the CIMTool RCP product for review, they display in the platform Generic Editor as uncolored plain text. Syntax highlighting would materially improve readability when inspecting generated output, particularly for the larger machine-generated files that profile authors and reviewers examine routinely.

The Eclipse-native mechanism for providing this highlighting is the TextMate grammar framework, delivered by the TM4E (TextMate for Eclipse) project. TM4E contributes a grammar engine and a bundled library of grammars (org.eclipse.tm4e.language_pack) that the platform Generic Editor uses to colorize content by content type, with no network access required at runtime. CIMTool does not currently integrate TM4E, so no syntax highlighting is available for any generated artifact type.

Describe the solution you'd like:

The desired outcome is that the CIMTool product bundles the official Eclipse TM4E runtime and binds the content types of generated text artifacts to the appropriate TM4E grammars, so that opening such a file in the Generic Editor produces syntax highlighting with no additional user configuration.

Preliminary integration work established the concrete approach. It is recorded here so it can be applied directly once the platform prerequisite (see Important below) is met:

1. Add the official Eclipse TM4E bundles to CIMTool.product: org.eclipse.tm4e.core, org.eclipse.tm4e.registry, org.eclipse.tm4e.ui, org.eclipse.tm4e.languageconfiguration, and org.eclipse.tm4e.language_pack. The grammar library is contained within org.eclipse.tm4e.language_pack and is air-gap safe (in-bundle grammars, no runtime fetch).

2. In CIMToolPlugin/plugin.xml, for each artifact file type, contribute three coordinated extensions:
   a. A content type under org.eclipse.core.contenttype.contentTypes declaring the file extension(s).
   b. A scopeNameContentTypeBinding under org.eclipse.tm4e.registry.grammars mapping that content type to the grammar scope provided by org.eclipse.tm4e.language_pack.
   c. An editorContentTypeBinding under org.eclipse.ui.editors associating that content type with the Generic Editor (org.eclipse.ui.genericeditor.GenericEditor).

   • No Require-Bundle entry in CIMToolPlugin/META-INF/MANIFEST.MF is required for these contributions, as they are resolved by the extension registry rather than by the bundle classpath.

3. The grammar scopes observed in the preliminary language_pack were lngpck.source.json, lngpck.source.java, lngpck.source.cs, lngpck.source.sql, lngpck.source.python, and lngpck.source.cpp. The exact scope identifiers and the set of available grammars vary between language_pack versions and should be re-verified during development against the grammar set that ships with the target platform.

4. A committed target platform definition (.target) pinning the TM4E bundles and their third-party dependencies should accompany this work, so that the product build resolves TM4E deterministically rather than depending on bundles installed in an individual developer's running IDE. Preliminary work confirmed that building against the live "Running Platform" is fragile and is a source of non-reproducible export behaviour.

Important — this enhancement is gated on a platform upgrade. Preliminary investigation established that no TM4E version simultaneously runs on the current CIMTool platform (Eclipse 2023-06 / 4.28) and is free of a blocking runtime defect:

• The TM4E release aligned with Eclipse 2023-06 (the 0.7.1 line; component bundles org.eclipse.tm4e.core 0.5.4, org.eclipse.tm4e.ui 0.6.4, org.eclipse.tm4e.language_pack 0.2.0) installs and loads its grammars, but throws java.lang.IllegalThreadStateException in org.eclipse.tm4e.core.model.TMModel.startTokenizerThread the moment the Generic Editor attempts to colorize a file. The exception aborts editor creation ("Failed to create the part's controls"), making this line unsuitable for a shipped product.

• The current TM4E release (0.17.2) resolves the tokenizer defect but requires a JavaSE-21 execution environment and the third-party bundles org.jcodings, org.joni, and org.snakeyaml.engine, none of which are present on the 2023-06 platform (which predates the Java 21 baseline). PDE validation reports these as unsatisfiable constraints, and the bundles cannot resolve.

The integration should therefore be undertaken in conjunction with, or following, a migration of CIMTool to a Java-21 Eclipse release (2024-09 / 4.33 or newer). At that point the current TM4E line integrates cleanly using the approach above, and a YAML grammar — absent from language_pack 0.2.0 but present in current packs (which depend on org.snakeyaml.engine) — becomes available. The timing of this enhancement is thus best aligned with the broader platform-currency upgrade rather than pursued in isolation.

Describe alternatives you've considered:

1. Vendoring individual TextMate grammars directly into CIMToolPlugin and registering them via the TM4E <grammar> extension, decoupling specific grammars from language_pack. This still requires a working TM4E runtime and therefore does not bypass the platform constraint; it would only be relevant for grammars absent from the chosen pack (for example, a YAML grammar on the older 0.2.0 pack). Each vendored grammar would be a third-party asset requiring license vetting and inclusion in the vendored-libraries documentation.

2. The community "Language Pack for Eclipse" (howlger), which packages Visual Studio Code–derived grammars and is distributed from its own GitHub-hosted update site. Set aside in favour of the official Eclipse org.eclipse.tm4e.language_pack for provenance, maintenance, and supply-chain reasons appropriate to critical-infrastructure deployment.

3. Retaining plain-text display (no action). This is the acceptable interim state while the platform constraint stands.

Additional context:

Preliminary investigation reference material follows.

TM4E 0.7.1 component bundle versions (the line compatible with Eclipse 2023-06):

Bundle	Version
org.eclipse.tm4e.core	0.5.4
org.eclipse.tm4e.languageconfiguration	0.5.5
org.eclipse.tm4e.registry	0.6.3
org.eclipse.tm4e.ui	0.6.4
org.eclipse.tm4e.markdown	0.5.1
org.eclipse.tm4e.language_pack	0.2.0

Blocking runtime defect on the 0.7.1 line — thrown on opening a grammar-bound file in the Generic Editor:

java.lang.IllegalThreadStateException
    at java.base/java.lang.Thread.start(Thread.java:1567)
    at org.eclipse.tm4e.core.model.TMModel.startTokenizerThread(TMModel.java:249)
    at org.eclipse.tm4e.core.model.TMModel.addModelTokensChangedListener(TMModel.java:223)
    at org.eclipse.tm4e.ui.text.TMPresentationReconciler$InternalListener.textChanged(...)
    ...
    at org.eclipse.ui.internal.genericeditor.folding.IndentFoldingStrategy.reconcile(...)
    at org.eclipse.ui.internal.genericeditor.ExtensionBasedTextEditor.createPartControl(...)

Unsatisfiable constraints on the current 0.17.2 line under the 2023-06 / pre-21 platform, as reported by PDE validation:

org.eclipse.tm4e.core
  Missing Constraint: Require-Bundle: org.jcodings; bundle-version="[1.0.63,2.0.0)"
  Missing Constraint: Require-Bundle: org.joni; bundle-version="[2.2.6,3.0.0)"
  Missing Constraint: Require-Bundle: org.snakeyaml.engine; bundle-version="[2.10.0,3.0.0)"
  Missing Constraint: Require-Capability: osgi.ee; filter="(&(osgi.ee=JavaSE)(version=21))"
org.eclipse.tm4e.languageconfiguration
  Missing Constraint: Require-Capability: osgi.ee; filter="(&(osgi.ee=JavaSE)(version=21))"
org.eclipse.tm4e.registry
  Missing Constraint: Require-Capability: osgi.ee; filter="(&(osgi.ee=JavaSE)(version=21))"
org.eclipse.tm4e.ui
  Missing Constraint: Require-Capability: osgi.ee; filter="(&(osgi.ee=JavaSE)(version=21))"

Reference links:

• Eclipse TM4E project: https://projects.eclipse.org/projects/technology.tm4e
• TM4E version-pinned update site used in preliminary work: https://download.eclipse.org/tm4e/releases/0.7.1
• Eclipse Generic Editor and content-type binding overview: https://www.eclipse.org/eclipse/news/4.7/platform.php

Conclusion for future planning: the integration design is understood and low-risk to apply; the sole blocker is the platform. This enhancement should be scheduled as a dependent item of a CIMTool migration to a Java-21 Eclipse release, and re-validated against the TM4E and language_pack versions current at that time.