tftsr-devops_investigation/node_modules/micromark-util-subtokenize/readme.md
Shaun Arman 8839075805 feat: initial implementation of TFTSR IT Triage & RCA application
Implements Phases 1-8 of the TFTSR implementation plan.

Rust backend (Tauri 2.x, src-tauri/):
- Multi-provider AI: OpenAI-compatible, Anthropic, Gemini, Mistral, Ollama
- PII detection engine: 11 regex patterns with overlap resolution
- SQLCipher AES-256 encrypted database with 10 versioned migrations
- 28 Tauri IPC commands for triage, analysis, document, and system ops
- Ollama: hardware probe, model recommendations, pull/delete with events
- RCA and blameless post-mortem Markdown document generators
- PDF export via printpdf
- Audit log: SHA-256 hash of every external data send
- Integration stubs for Confluence, ServiceNow, Azure DevOps (v0.2)

Frontend (React 18 + TypeScript + Vite, src/):
- 9 pages: full triage workflow NewIssue→LogUpload→Triage→Resolution→RCA→Postmortem→History+Settings
- 7 components: ChatWindow, TriageProgress, PiiDiffViewer, DocEditor, HardwareReport, ModelSelector, UI primitives
- 3 Zustand stores: session, settings (persisted), history
- Type-safe tauriCommands.ts matching Rust backend types exactly
- 8 IT domain system prompts (Linux, Windows, Network, K8s, DB, Virt, HW, Obs)

DevOps:
- .woodpecker/test.yml: rustfmt, clippy, cargo test, tsc, vitest on every push
- .woodpecker/release.yml: linux/amd64 + linux/arm64 builds, Gogs release upload

Verified:
- cargo check: zero errors
- tsc --noEmit: zero errors
- vitest run: 13/13 unit tests passing

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-14 22:36:25 -05:00

182 lines
4.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# micromark-util-subtokenize
[![Build][build-badge]][build]
[![Coverage][coverage-badge]][coverage]
[![Downloads][downloads-badge]][downloads]
[![Size][bundle-size-badge]][bundle-size]
[![Sponsors][sponsors-badge]][opencollective]
[![Backers][backers-badge]][opencollective]
[![Chat][chat-badge]][chat]
[micromark][] utility to tokenize subtokens.
## Contents
* [What is this?](#what-is-this)
* [When should I use this?](#when-should-i-use-this)
* [Install](#install)
* [Use](#use)
* [API](#api)
* [`subtokenize(events)`](#subtokenizeevents)
* [Types](#types)
* [Compatibility](#compatibility)
* [Security](#security)
* [Contribute](#contribute)
* [License](#license)
## What is this?
This package exposes a micromark internal that you probably dont need.
## When should I use this?
This package might be useful when you are making your own micromark extensions.
## Install
This package is [ESM only][esm].
In Node.js (version 16+), install with [npm][]:
```sh
npm install micromark-util-subtokenize
```
In Deno with [`esm.sh`][esmsh]:
```js
import {subtokenize} from 'https://esm.sh/micromark-util-subtokenize@1'
```
In browsers with [`esm.sh`][esmsh]:
```html
<script type="module">
import {subtokenize} from 'https://esm.sh/micromark-util-subtokenize@1?bundle'
</script>
```
## Use
```js
import {subtokenize} from 'micromark-util-subtokenize'
/**
* Content is transparent: its parsed right now. That way, definitions are also
* parsed right now: before text in paragraphs (specifically, media) are parsed.
*
* @type {Resolver}
*/
function resolveContent(events) {
subtokenize(events)
return events
}
```
## API
This module exports the identifiers [`subtokenize`][api-subtokenize].
There is no default export.
### `subtokenize(events)`
Tokenize subcontent.
###### Parameters
* `events` (`Array<Event>`)
— list of events
###### Returns
Whether subtokens were found (`boolean`).
## Types
This package is fully typed with [TypeScript][].
It exports no additional types.
## Compatibility
Projects maintained by the unified collective are compatible with maintained
versions of Node.js.
When we cut a new major release, we drop support for unmaintained versions of
Node.
This means we try to keep the current release line,
`micromark-util-subtokenize@2`, compatible with Node.js 16.
This package works with `micromark@3`.
## Security
This package is safe.
See [`security.md`][securitymd] in [`micromark/.github`][health] for how to
submit a security report.
## Contribute
See [`contributing.md`][contributing] in [`micromark/.github`][health] for ways
to get started.
See [`support.md`][support] for ways to get help.
This project has a [code of conduct][coc].
By interacting with this repository, organisation, or community you agree to
abide by its terms.
## License
[MIT][license] © [Titus Wormer][author]
<!-- Definitions -->
[api-subtokenize]: #subtokenizeevents
[author]: https://wooorm.com
[backers-badge]: https://opencollective.com/unified/backers/badge.svg
[build]: https://github.com/micromark/micromark/actions
[build-badge]: https://github.com/micromark/micromark/workflows/main/badge.svg
[bundle-size]: https://bundlejs.com/?q=micromark-util-subtokenize
[bundle-size-badge]: https://img.shields.io/badge/dynamic/json?label=minzipped%20size&query=$.size.compressedSize&url=https://deno.bundlejs.com/?q=micromark-util-subtokenize
[chat]: https://github.com/micromark/micromark/discussions
[chat-badge]: https://img.shields.io/badge/chat-discussions-success.svg
[coc]: https://github.com/micromark/.github/blob/main/code-of-conduct.md
[contributing]: https://github.com/micromark/.github/blob/main/contributing.md
[coverage]: https://codecov.io/github/micromark/micromark
[coverage-badge]: https://img.shields.io/codecov/c/github/micromark/micromark.svg
[downloads]: https://www.npmjs.com/package/micromark-util-subtokenize
[downloads-badge]: https://img.shields.io/npm/dm/micromark-util-subtokenize.svg
[esm]: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c
[esmsh]: https://esm.sh
[health]: https://github.com/micromark/.github
[license]: https://github.com/micromark/micromark/blob/main/license
[micromark]: https://github.com/micromark/micromark
[npm]: https://docs.npmjs.com/cli/install
[opencollective]: https://opencollective.com/unified
[securitymd]: https://github.com/micromark/.github/blob/main/security.md
[sponsors-badge]: https://opencollective.com/unified/sponsors/badge.svg
[support]: https://github.com/micromark/.github/blob/main/support.md
[typescript]: https://www.typescriptlang.org