Updates.
This commit is contained in:
157
test/README.md
Normal file
157
test/README.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# Unified Test Harness for Terrace
|
||||
|
||||
This directory contains the unified test harness for all Terrace language implementations. The harness automatically discovers and runs tests across all supported languages: JavaScript, C, Python, Go, and Rust.
|
||||
|
||||
## Features
|
||||
|
||||
- **Cross-language testing**: Run the same tests across all language implementations
|
||||
- **Automatic test discovery**: Finds all `.tce` test files in the project
|
||||
- **Flexible test definitions**: Tests are defined in human-readable `.tce` files
|
||||
- **Build integration**: Automatically builds required binaries (C test runner)
|
||||
- **Comprehensive reporting**: Detailed test results with pass/fail status
|
||||
|
||||
## Test File Format
|
||||
|
||||
Test files use the Terrace format with the following structure:
|
||||
|
||||
```terrace
|
||||
#schema test
|
||||
|
||||
describe Test Suite Name
|
||||
it Test description
|
||||
key test-key
|
||||
packages js c python go rust
|
||||
input
|
||||
test input data
|
||||
more input lines
|
||||
output
|
||||
expected output line 1
|
||||
expected output line 2
|
||||
```
|
||||
|
||||
### Fields
|
||||
|
||||
- `describe`: Defines a test suite
|
||||
- `it`: Defines a test case
|
||||
- `key`: The test identifier passed to language implementations
|
||||
- `packages`: Space-separated list of packages to test (js, c, python, go, rust)
|
||||
- `input`: Test input data (optional)
|
||||
- `output`: Expected output (optional for Go/Rust which use different testing frameworks)
|
||||
|
||||
## Usage
|
||||
|
||||
### Run all tests
|
||||
|
||||
```bash
|
||||
npm test
|
||||
# or
|
||||
npm run test:unified
|
||||
```
|
||||
|
||||
### Run specific test
|
||||
|
||||
```bash
|
||||
node test/test-runner.js <test-key>
|
||||
```
|
||||
|
||||
### Run tests for specific packages
|
||||
|
||||
```bash
|
||||
PACKAGES="js python" node test/test-runner.js <test-key>
|
||||
```
|
||||
|
||||
## Supported Test Keys
|
||||
|
||||
### JavaScript
|
||||
|
||||
- `linedata:basic` - Basic line data parsing
|
||||
- `linedata:tabs` - Tab-based indentation
|
||||
- `linedata:head-tail` - Head/tail parsing
|
||||
- `new-api:basic` - Basic new API functionality
|
||||
- `new-api:hierarchical` - Hierarchical document parsing
|
||||
- `new-api:functional` - Functional API features
|
||||
- `new-api:node-methods` - Node method testing
|
||||
- `new-api:inconsistent-indentation` - Arbitrary nesting
|
||||
|
||||
### C
|
||||
|
||||
- `linedata:basic` - Basic line data parsing
|
||||
- `linedata:tabs` - Tab-based indentation
|
||||
- `linedata:head-tail` - Head/tail parsing
|
||||
- `new-api:basic` - Basic new API functionality
|
||||
- `new-api:hierarchical` - Hierarchical document parsing
|
||||
- `new-api:string-views` - String view functionality
|
||||
- `new-api:legacy-compat` - Legacy API compatibility
|
||||
|
||||
### Python
|
||||
|
||||
- `linedata:basic` - Basic line data parsing
|
||||
- `linedata:tabs` - Tab-based indentation
|
||||
- `linedata:head-tail` - Head/tail parsing
|
||||
- `new-api:basic` - Basic new API functionality
|
||||
- `new-api:hierarchical` - Hierarchical document parsing
|
||||
- `new-api:functional` - Functional API features
|
||||
- `new-api:node-methods` - Node method testing
|
||||
- `new-api:readers` - Reader utilities
|
||||
|
||||
### Go
|
||||
|
||||
- `TestTerraceDocument` - Document parsing tests
|
||||
- `TestTerraceNode` - Node functionality tests
|
||||
|
||||
### Rust
|
||||
|
||||
- `test_basic_parsing` - Basic parsing functionality
|
||||
- `test_navigation_methods` - Navigation and traversal
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
1. Create a new `.tce` file in the `test/` directory or modify existing ones
|
||||
2. Define test suites and cases using the Terrace format
|
||||
3. Specify which packages should run the test
|
||||
4. Add the test key to the appropriate language implementation's test runner
|
||||
5. Update the `supports` array in `test-runner.js` if needed
|
||||
|
||||
## Test Output Format
|
||||
|
||||
Each language implementation should output test results in a consistent format for comparison. The expected format varies by test type:
|
||||
|
||||
- **LineData tests**: `| level X | indent Y | offsetHead Z | offsetTail W | line TEXT |`
|
||||
- **New API tests**: `| level X | head "HEAD" | tail "TAIL" | content "CONTENT" |`
|
||||
- **Node method tests**: `Node: head="HEAD", tail="TAIL", isEmpty=BOOL, is(NAME)=BOOL`
|
||||
- **Go/Rust tests**: Simple "Test passed" message
|
||||
|
||||
## Integration with CI/CD
|
||||
|
||||
The test harness is designed to work with CI/CD systems:
|
||||
|
||||
- Returns exit code 0 on success, 1 on failure
|
||||
- Provides detailed output for debugging
|
||||
- Can be run in parallel across different environments
|
||||
- Supports selective test execution
|
||||
|
||||
## Dependencies
|
||||
|
||||
The test harness requires:
|
||||
|
||||
- Node.js for the test runner
|
||||
- Build tools for each language (make, go, cargo, etc.)
|
||||
- All language implementations to be properly set up
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **C tests fail**: Ensure the C test runner is built with `make` in `packages/c/`
|
||||
2. **Go tests fail**: Ensure Go modules are properly initialized
|
||||
3. **Rust tests fail**: Ensure Cargo dependencies are installed
|
||||
4. **Python tests fail**: Ensure Python dependencies are installed
|
||||
5. **Test discovery fails**: Check that `.tce` files have correct syntax
|
||||
|
||||
### Debug Mode
|
||||
|
||||
Run with verbose output:
|
||||
|
||||
```bash
|
||||
DEBUG=1 node test/test-runner.js
|
||||
```
|
||||
182
test/comprehensive.test.tce
Normal file
182
test/comprehensive.test.tce
Normal file
@@ -0,0 +1,182 @@
|
||||
#schema test
|
||||
|
||||
describe New API Basic Tests
|
||||
it Parses simple hierarchical structure
|
||||
key new-api:basic
|
||||
input
|
||||
title My Document
|
||||
description
|
||||
This is a description
|
||||
With multiple lines
|
||||
section Getting Started
|
||||
step Install the package
|
||||
step Run the tests
|
||||
output
|
||||
| level 0 | head "title" | tail "My Document" | content "title My Document" |
|
||||
| level 0 | head "description" | tail "" | content "description" |
|
||||
| level 1 | head "This" | tail "is a description" | content "This is a description" |
|
||||
| level 1 | head "With" | tail "multiple lines" | content "With multiple lines" |
|
||||
| level 0 | head "section" | tail "Getting Started" | content "section Getting Started" |
|
||||
| level 1 | head "step" | tail "Install the package" | content "step Install the package" |
|
||||
| level 1 | head "step" | tail "Run the tests" | content "step Run the tests" |
|
||||
|
||||
it Handles empty lines and whitespace
|
||||
key new-api:empty-lines
|
||||
input
|
||||
item1 value1
|
||||
|
||||
item2 value2
|
||||
nested under item2
|
||||
output
|
||||
| level 0 | head "item1" | tail "value1" | content "item1 value1" |
|
||||
| level 0 | head "item2" | tail "value2" | content "item2 value2" |
|
||||
| level 1 | head "nested" | tail "under item2" | content "nested under item2" |
|
||||
|
||||
it Parses complex nested structures
|
||||
key new-api:hierarchical
|
||||
input
|
||||
config
|
||||
database
|
||||
host localhost
|
||||
port 5432
|
||||
server
|
||||
port 3000
|
||||
host 0.0.0.0
|
||||
feature_flags
|
||||
debug true
|
||||
logging false
|
||||
output
|
||||
| level 0 | head "config" | tail "" | content "config" |
|
||||
| level 1 | head "database" | tail "" | content "database" |
|
||||
| level 2 | head "host" | tail "localhost" | content "host localhost" |
|
||||
| level 2 | head "port" | tail "5432" | content "port 5432" |
|
||||
| level 1 | head "server" | tail "" | content "server" |
|
||||
| level 2 | head "port" | tail "3000" | content "port 3000" |
|
||||
| level 2 | head "host" | tail "0.0.0.0" | content "host 0.0.0.0" |
|
||||
| level 0 | head "feature_flags" | tail "" | content "feature_flags" |
|
||||
| level 1 | head "debug" | tail "true" | content "debug true" |
|
||||
| level 1 | head "logging" | tail "false" | content "logging false" |
|
||||
|
||||
describe Node Methods Tests
|
||||
it Tests node properties and methods
|
||||
key new-api:node-methods
|
||||
input
|
||||
title Test Document
|
||||
empty_line
|
||||
|
||||
regular_line With some content
|
||||
multi word content
|
||||
output
|
||||
Node: head="title", tail="Test Document", isEmpty=false, is(title)=true
|
||||
content="title Test Document", raw(0)="title Test Document", lineNumber=1
|
||||
Node: head="empty_line", tail="", isEmpty=true, is(title)=false
|
||||
content="empty_line", raw(0)="empty_line", lineNumber=2
|
||||
Node: head="regular_line", tail="With some content", isEmpty=false, is(title)=false
|
||||
content="regular_line With some content", raw(0)="regular_line With some content", lineNumber=4
|
||||
Node: head="multi", tail="word content", isEmpty=false, is(title)=false
|
||||
content="multi word content", raw(0)="multi word content", lineNumber=5
|
||||
|
||||
describe Functional API Tests
|
||||
it Tests filtering and finding nodes
|
||||
key new-api:functional
|
||||
input
|
||||
config
|
||||
database
|
||||
host localhost
|
||||
port 5432
|
||||
name myapp
|
||||
server
|
||||
port 3000
|
||||
host 0.0.0.0
|
||||
feature_flags
|
||||
debug true
|
||||
logging false
|
||||
output
|
||||
Found 2 config sections
|
||||
Found feature flags section
|
||||
|
||||
describe Inconsistent Indentation Tests
|
||||
it Handles arbitrary nesting levels
|
||||
key new-api:inconsistent-indentation
|
||||
input
|
||||
level1
|
||||
level2
|
||||
level5
|
||||
level3
|
||||
level6
|
||||
level2
|
||||
output
|
||||
| level 0 | head "level1" | tail "" |
|
||||
| level 1 | head "level2" | tail "" |
|
||||
| level 4 | head "level5" | tail "" |
|
||||
| level 2 | head "level3" | tail "" |
|
||||
| level 5 | head "level6" | tail "" |
|
||||
| level 1 | head "level2" | tail "" |
|
||||
|
||||
describe Reader Utilities Tests
|
||||
it Tests different reader implementations
|
||||
key new-api:readers
|
||||
input
|
||||
item1 value1
|
||||
nested1 nested_value1
|
||||
nested2 nested_value2
|
||||
item2 value2
|
||||
output
|
||||
item1: value1
|
||||
nested1: nested_value1
|
||||
nested2: nested_value2
|
||||
item2: value2
|
||||
|
||||
describe Node Methods Tests
|
||||
it Tests node properties and methods
|
||||
key new-api:node-methods
|
||||
input
|
||||
multi word content
|
||||
output
|
||||
describe Functional API Tests
|
||||
it Tests filtering and finding nodes
|
||||
key new-api:functional
|
||||
input
|
||||
config
|
||||
database
|
||||
host localhost
|
||||
port 5432
|
||||
server
|
||||
port 3000
|
||||
host 0.0.0.0
|
||||
feature_flags
|
||||
enable_new_ui true
|
||||
enable_beta_features false
|
||||
output
|
||||
Found feature flags section
|
||||
Found 2 config sections
|
||||
|
||||
describe Legacy Compatibility Tests
|
||||
it Tests legacy API compatibility
|
||||
key new-api:legacy-compat
|
||||
input
|
||||
config
|
||||
database localhost
|
||||
server 3000
|
||||
feature_flags
|
||||
debug true
|
||||
output
|
||||
Found config section using legacy API
|
||||
Config item: head starts with 'd', tail='localhost'
|
||||
Config item: head starts with 's', tail='3000'
|
||||
|
||||
describe Content Method Tests
|
||||
it Content method uses offsetHead correctly
|
||||
key new-api:content-method
|
||||
input
|
||||
title My Document Title
|
||||
section Introduction
|
||||
paragraph This is a paragraph with content
|
||||
code
|
||||
console.log("Hello World")
|
||||
output
|
||||
| level 0 | head "title" | tail "My Document Title" | content "title My Document Title" |
|
||||
| level 0 | head "section" | tail "Introduction" | content "section Introduction" |
|
||||
| level 1 | head "paragraph" | tail "This is a paragraph with content" | content "paragraph This is a paragraph with content" |
|
||||
| level 1 | head "code" | tail "" | content "code" |
|
||||
| level 2 | head "console.log("Hello" | tail "World")" | content "console.log("Hello World")" |
|
||||
@@ -5,53 +5,82 @@ import { useDocument } from '@terrace-lang/js'
|
||||
import { createFileReader } from '@terrace-lang/js/readers/node-readline'
|
||||
|
||||
export async function loadTestMap(path) {
|
||||
const { next, level, head, tail, line, match } = useDocument(createFileReader(path))
|
||||
const reader = createFileReader(path)
|
||||
const doc = useDocument(reader)
|
||||
|
||||
const descriptions = {}
|
||||
let currentSuite = null
|
||||
let currentTest = null
|
||||
|
||||
while (await next()) {
|
||||
if (!head() || match('#schema')) continue
|
||||
const tests = descriptions[tail()] = []
|
||||
for await (const node of doc) {
|
||||
if (node.head === '#schema') {
|
||||
// Skip schema declarations
|
||||
continue
|
||||
}
|
||||
|
||||
const rootLevel = level()
|
||||
while (await next(rootLevel)) {
|
||||
if (!match('it')) continue
|
||||
if (node.head === 'describe') {
|
||||
currentSuite = node.tail
|
||||
descriptions[currentSuite] = []
|
||||
continue
|
||||
}
|
||||
|
||||
const test = { it: tail(), packages: [], key: '', input: [], output: [] }
|
||||
tests.push(test)
|
||||
if (node.head === 'it' && currentSuite) {
|
||||
currentTest = {
|
||||
it: node.tail,
|
||||
packages: [],
|
||||
key: '',
|
||||
input: '',
|
||||
output: ''
|
||||
}
|
||||
descriptions[currentSuite].push(currentTest)
|
||||
|
||||
const testLevel = level()
|
||||
while (await next(testLevel)) {
|
||||
if (match('key')) test.key = tail()
|
||||
if (match('packages')) test.packages = tail().split(' ')
|
||||
// Collect test properties
|
||||
let collectingInput = false
|
||||
let collectingOutput = false
|
||||
let inputLevel = 0
|
||||
|
||||
const propertyLevel = level()
|
||||
if (match('input')) {
|
||||
// TODO: Find a way to handle newlines better.
|
||||
if (tail()) {
|
||||
test.input = tail()
|
||||
continue
|
||||
for await (const child of node.children()) {
|
||||
if (child.head === 'key') {
|
||||
currentTest.key = child.tail
|
||||
} else if (child.head === 'packages') {
|
||||
currentTest.packages = child.tail.split(' ')
|
||||
} else if (child.head === 'input') {
|
||||
collectingInput = true
|
||||
collectingOutput = false
|
||||
inputLevel = child.level
|
||||
// If input has content on the same line
|
||||
if (child.tail) {
|
||||
currentTest.input = child.tail
|
||||
}
|
||||
|
||||
while (await next(propertyLevel)) test.input.push(line(propertyLevel + 1))
|
||||
test.input = test.input.join('\n').trimEnd()
|
||||
}
|
||||
|
||||
if (match('output')) {
|
||||
while (await next(propertyLevel)) test.output.push(line(propertyLevel + 1))
|
||||
} else if (child.head === 'output') {
|
||||
collectingInput = false
|
||||
collectingOutput = true
|
||||
// If output has content on the same line
|
||||
if (child.tail) {
|
||||
currentTest.output = child.tail
|
||||
}
|
||||
} else if (collectingInput) {
|
||||
// Collect input lines
|
||||
currentTest.input += (currentTest.input ? '\n' : '') + child.raw(inputLevel + 1)
|
||||
} else if (collectingOutput) {
|
||||
// Collect output lines
|
||||
currentTest.output += (currentTest.output ? '\n' : '') + child.content
|
||||
}
|
||||
}
|
||||
|
||||
test.input = test.input
|
||||
.replaceAll('\\n', '\n')
|
||||
.replaceAll('\\t', '\t')
|
||||
.replaceAll('\\s', ' ')
|
||||
// Process escape sequences
|
||||
if (typeof currentTest.input === 'string') {
|
||||
currentTest.input = currentTest.input
|
||||
.replaceAll('\\n', '\n')
|
||||
.replaceAll('\\t', '\t')
|
||||
.replaceAll('\\s', ' ')
|
||||
}
|
||||
|
||||
test.output = test.output.join('\n').trimEnd()
|
||||
.replaceAll('\\n', '\n')
|
||||
.replaceAll('\\t', '\t')
|
||||
.replaceAll('\\s', ' ')
|
||||
if (typeof currentTest.output === 'string') {
|
||||
currentTest.output = currentTest.output.trimEnd()
|
||||
}
|
||||
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
import { loadTestMap, defineTests } from './helpers.js'
|
||||
|
||||
defineTests(await loadTestMap('./lineData.test.tce'))
|
||||
@@ -3,28 +3,24 @@
|
||||
describe LineData
|
||||
it Handles a blank line at indent level 0
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input \n
|
||||
output
|
||||
| level 0 | indent | offsetHead 0 | offsetTail 0 | line |
|
||||
|
||||
it Handles a blank line with a single space
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input \s
|
||||
output
|
||||
| level 1 | indent | offsetHead 1 | offsetTail 1 | line |
|
||||
|
||||
it Handles a blank line with two spaces
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input \s\s
|
||||
output
|
||||
| level 2 | indent | offsetHead 2 | offsetTail 2 | line |
|
||||
|
||||
it Handles a normal line at indent level 0
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input
|
||||
line 1
|
||||
output
|
||||
@@ -32,7 +28,6 @@ describe LineData
|
||||
|
||||
it Handles a normal line at indent level 1
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input
|
||||
line 1
|
||||
output
|
||||
@@ -40,7 +35,6 @@ describe LineData
|
||||
|
||||
it Handles a normal line at indent level 2
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input
|
||||
line 1
|
||||
output
|
||||
@@ -48,7 +42,6 @@ describe LineData
|
||||
|
||||
it Handles a normal line at indent level 1 indented with tabs
|
||||
key linedata:tabs
|
||||
packages c js python
|
||||
input
|
||||
\tline 1
|
||||
output
|
||||
@@ -56,7 +49,6 @@ describe LineData
|
||||
|
||||
it Handles a normal line at indent level 2 indented with tabs
|
||||
key linedata:tabs
|
||||
packages c js python
|
||||
input
|
||||
\t\tline 1
|
||||
output
|
||||
@@ -64,7 +56,6 @@ describe LineData
|
||||
|
||||
it Nests a normal line under a preceding normal line
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input
|
||||
line 1
|
||||
line 2
|
||||
@@ -74,7 +65,6 @@ describe LineData
|
||||
|
||||
it Nests multiple normal lines under a preceding normal line
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
input
|
||||
line 1
|
||||
line 2
|
||||
@@ -88,7 +78,6 @@ describe LineData
|
||||
|
||||
it Does not nest an empty line under a preceding normal line
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
comment Two newlines are needed here. A single newline will look to readline as if the input is finished.
|
||||
input line 1\n\n
|
||||
output
|
||||
@@ -97,7 +86,6 @@ describe LineData
|
||||
|
||||
it Does not nest multiple empty lines under a preceding normal line
|
||||
key linedata:basic
|
||||
packages c js python
|
||||
comment Four newlines are needed here. A single newline will look to readline as if the input is finished.
|
||||
input line 1\n\n\n\n
|
||||
output
|
||||
@@ -108,7 +96,6 @@ describe LineData
|
||||
|
||||
it Handles head and tail matching for lines with head and tail
|
||||
key linedata:head-tail
|
||||
packages c js python
|
||||
input
|
||||
head1 tail1 tail2 tail3
|
||||
output
|
||||
@@ -124,7 +111,6 @@ describe LineData
|
||||
|
||||
it Handles head and tail matching for lines with head and trailing space
|
||||
key linedata:head-tail
|
||||
packages c js python
|
||||
input head1 \n
|
||||
output
|
||||
| head head1 | tail |
|
||||
|
||||
@@ -2,7 +2,13 @@
|
||||
"name": "@terrace-lang/test",
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"test": "NODE_OPTIONS=--experimental-vm-modules NODE_NO_WARNINGS=1 jest"
|
||||
"test:unified": "node test-runner.js",
|
||||
"test:specific": "node test-runner.js",
|
||||
"test:python": "node test-runner.js --lang python",
|
||||
"test:js": "node test-runner.js --lang js",
|
||||
"test:c": "node test-runner.js --lang c",
|
||||
"test:go": "node test-runner.js --lang go",
|
||||
"test:rust": "node test-runner.js --lang rust"
|
||||
},
|
||||
"dependencies": {
|
||||
"@terrace-lang/c": "workspace:*",
|
||||
@@ -12,4 +18,4 @@
|
||||
"devDependencies": {
|
||||
"chai": "^4.3.7"
|
||||
}
|
||||
}
|
||||
}
|
||||
349
test/test-runner.js
Executable file
349
test/test-runner.js
Executable file
@@ -0,0 +1,349 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { execSync } from 'node:child_process'
|
||||
import fs from 'node:fs/promises'
|
||||
import path from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url))
|
||||
|
||||
// Language implementations and their test commands
|
||||
const LANGUAGES = {
|
||||
js: {
|
||||
command: 'node test/index.js',
|
||||
cwd: path.join(__dirname, '..', 'packages', 'js'),
|
||||
},
|
||||
c: {
|
||||
command: './test/test-runner',
|
||||
cwd: path.join(__dirname, '..', 'packages', 'c'),
|
||||
buildCommand: 'make',
|
||||
},
|
||||
python: {
|
||||
command: 'python3 test/index.py',
|
||||
cwd: path.join(__dirname, '..', 'packages', 'python'),
|
||||
},
|
||||
go: {
|
||||
command: 'go run test/test-runner.go',
|
||||
cwd: path.join(__dirname, '..', 'packages', 'go'),
|
||||
},
|
||||
rust: {
|
||||
command: 'cargo run --bin test-runner --',
|
||||
cwd: path.join(__dirname, '..', 'packages', 'rust'),
|
||||
}
|
||||
}
|
||||
|
||||
// Test result tracking
|
||||
let totalTests = 0
|
||||
let passedTests = 0
|
||||
let failedTests = 0
|
||||
const failures = []
|
||||
|
||||
async function runTest(language, testName, input) {
|
||||
const langConfig = LANGUAGES[language]
|
||||
if (!langConfig) {
|
||||
throw new Error(`Unknown language: ${language}`)
|
||||
}
|
||||
|
||||
// Build if necessary
|
||||
if (langConfig.buildCommand) {
|
||||
try {
|
||||
execSync(langConfig.buildCommand, {
|
||||
cwd: langConfig.cwd,
|
||||
stdio: 'pipe'
|
||||
})
|
||||
} catch (error) {
|
||||
return {
|
||||
error: `Build failed: ${error.message}`,
|
||||
success: false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
let command
|
||||
// All languages now take input from stdin and produce comparable output
|
||||
command = `${langConfig.command} ${testName}`
|
||||
const result = execSync(command, {
|
||||
cwd: langConfig.cwd,
|
||||
input: input || '',
|
||||
encoding: 'utf8',
|
||||
timeout: 10000 // 10 second timeout
|
||||
})
|
||||
return { output: result.trim(), success: true }
|
||||
} catch (error) {
|
||||
return {
|
||||
error: error.message,
|
||||
stderr: error.stderr?.toString() || '',
|
||||
success: false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function findTestFiles() {
|
||||
const allTceFiles = []
|
||||
|
||||
async function scanDirectory(dir) {
|
||||
try {
|
||||
const entries = await fs.readdir(dir, { withFileTypes: true })
|
||||
|
||||
for (const entry of entries) {
|
||||
const fullPath = path.join(dir, entry.name)
|
||||
|
||||
if (entry.isDirectory() && !entry.name.startsWith('.') && entry.name !== 'node_modules') {
|
||||
await scanDirectory(fullPath)
|
||||
} else if (entry.isFile() && entry.name.endsWith('.tce')) {
|
||||
allTceFiles.push(fullPath)
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// Skip directories we can't read
|
||||
}
|
||||
}
|
||||
|
||||
await scanDirectory(path.join(__dirname, '..'))
|
||||
|
||||
// Filter to only test files (those containing #schema test)
|
||||
const testFiles = []
|
||||
for (const file of allTceFiles) {
|
||||
try {
|
||||
const content = await fs.readFile(file, 'utf8')
|
||||
if (content.includes('#schema test')) {
|
||||
testFiles.push(file)
|
||||
}
|
||||
} catch (error) {
|
||||
// Skip files we can't read
|
||||
}
|
||||
}
|
||||
|
||||
return testFiles
|
||||
}
|
||||
|
||||
async function loadTestMap(filePath) {
|
||||
// Import the helpers from the test directory
|
||||
const { loadTestMap: loadMap } = await import('./helpers.js')
|
||||
return await loadMap(filePath)
|
||||
}
|
||||
|
||||
async function runUnifiedTests(languageFilter = null) {
|
||||
console.log('🔍 Discovering test files...')
|
||||
const testFiles = await findTestFiles()
|
||||
console.log(`📁 Found ${testFiles.length} test file(s)`)
|
||||
|
||||
if (languageFilter) {
|
||||
console.log(`🎯 Filtering to languages: ${languageFilter.join(', ')}`)
|
||||
}
|
||||
|
||||
for (const testFile of testFiles) {
|
||||
console.log(`\n📄 Running tests from: ${path.relative(path.join(__dirname, '..'), testFile)}`)
|
||||
|
||||
try {
|
||||
const testMap = await loadTestMap(testFile)
|
||||
|
||||
for (const [suiteName, tests] of Object.entries(testMap)) {
|
||||
console.log(`\n🧪 Test Suite: ${suiteName}`)
|
||||
|
||||
for (const test of tests) {
|
||||
console.log(`\n 📝 ${test.it}`)
|
||||
totalTests++
|
||||
|
||||
const expectedOutput = test.output
|
||||
const input = test.input
|
||||
let packages = test.packages.length > 0 ? test.packages : Object.keys(LANGUAGES)
|
||||
|
||||
// Apply language filter if specified
|
||||
if (languageFilter) {
|
||||
packages = packages.filter(pkg => languageFilter.includes(pkg))
|
||||
}
|
||||
|
||||
// Skip test if no packages match the filter
|
||||
if (packages.length === 0) {
|
||||
console.log(` ⏭️ Skipped (no matching languages)`)
|
||||
continue
|
||||
}
|
||||
|
||||
let testPassed = true
|
||||
const results = {}
|
||||
|
||||
for (const pkg of packages) {
|
||||
if (!LANGUAGES[pkg]) {
|
||||
console.log(` ⚠️ Unknown package: ${pkg}`)
|
||||
continue
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await runTest(pkg, test.key, input)
|
||||
|
||||
if (result.skipped) {
|
||||
results[pkg] = { status: 'skipped' }
|
||||
continue
|
||||
}
|
||||
|
||||
if (result.success) {
|
||||
const actualOutput = result.output
|
||||
// Compare outputs for all languages
|
||||
if (actualOutput === expectedOutput) {
|
||||
results[pkg] = { status: 'passed', output: actualOutput }
|
||||
} else {
|
||||
results[pkg] = {
|
||||
status: 'failed',
|
||||
output: actualOutput,
|
||||
expected: expectedOutput
|
||||
}
|
||||
testPassed = false
|
||||
}
|
||||
} else {
|
||||
results[pkg] = {
|
||||
status: 'error',
|
||||
error: result.error,
|
||||
stderr: result.stderr
|
||||
}
|
||||
testPassed = false
|
||||
}
|
||||
} catch (error) {
|
||||
results[pkg] = { status: 'error', error: error.message }
|
||||
testPassed = false
|
||||
}
|
||||
}
|
||||
|
||||
// Display results
|
||||
for (const [pkg, result] of Object.entries(results)) {
|
||||
if (result.status === 'passed') {
|
||||
console.log(` ✅ ${pkg}: PASSED`)
|
||||
} else if (result.status === 'skipped') {
|
||||
console.log(` ⏭️ ${pkg}: SKIPPED`)
|
||||
} else if (result.status === 'failed') {
|
||||
console.log(` ❌ ${pkg}: FAILED`)
|
||||
console.log(` Expected: ${result.expected}`)
|
||||
console.log(` Got: ${result.output}`)
|
||||
} else if (result.status === 'error') {
|
||||
console.log(` 💥 ${pkg}: ERROR`)
|
||||
if (result.error) console.log(` Error: ${result.error}`)
|
||||
if (result.stderr) console.log(` Stderr: ${result.stderr}`)
|
||||
}
|
||||
}
|
||||
|
||||
if (testPassed) {
|
||||
passedTests++
|
||||
} else {
|
||||
failedTests++
|
||||
failures.push({
|
||||
suite: suiteName,
|
||||
test: test.it,
|
||||
file: testFile,
|
||||
results
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`❌ Error loading test file ${testFile}: ${error.message}`)
|
||||
failedTests++
|
||||
failures.push({
|
||||
file: testFile,
|
||||
error: error.message
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Summary
|
||||
console.log('\n' + '='.repeat(50))
|
||||
console.log('🏁 TEST SUMMARY')
|
||||
console.log('='.repeat(50))
|
||||
console.log(`Total Tests: ${totalTests}`)
|
||||
console.log(`Passed: ${passedTests}`)
|
||||
console.log(`Failed: ${failedTests}`)
|
||||
console.log(`Success Rate: ${totalTests > 0 ? ((passedTests / totalTests) * 100).toFixed(1) : 0}%`)
|
||||
|
||||
if (failures.length > 0) {
|
||||
console.log('\n❌ FAILURES:')
|
||||
for (const failure of failures) {
|
||||
console.log(` - ${failure.suite || 'File'}: ${failure.test || failure.file}`)
|
||||
}
|
||||
process.exit(1)
|
||||
} else {
|
||||
console.log('\n✅ All tests passed!')
|
||||
}
|
||||
}
|
||||
|
||||
async function runSpecificTest(testName, packages = null) {
|
||||
const targetPackages = packages ? packages.split(',') : Object.keys(LANGUAGES)
|
||||
|
||||
console.log(`🎯 Running specific test: ${testName}`)
|
||||
console.log(`📦 Target packages: ${targetPackages.join(', ')}`)
|
||||
|
||||
let testPassed = true
|
||||
const results = {}
|
||||
|
||||
for (const pkg of targetPackages) {
|
||||
if (!LANGUAGES[pkg]) {
|
||||
console.log(`⚠️ Unknown package: ${pkg}`)
|
||||
continue
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await runTest(pkg, testName)
|
||||
|
||||
if (result.skipped) {
|
||||
results[pkg] = { status: 'skipped' }
|
||||
} else if (result.success) {
|
||||
results[pkg] = { status: 'passed', output: result.output }
|
||||
} else {
|
||||
results[pkg] = { status: 'error', error: result.error, stderr: result.stderr }
|
||||
testPassed = false
|
||||
}
|
||||
} catch (error) {
|
||||
results[pkg] = { status: 'error', error: error.message }
|
||||
testPassed = false
|
||||
}
|
||||
}
|
||||
|
||||
// Display results
|
||||
for (const [pkg, result] of Object.entries(results)) {
|
||||
if (result.status === 'passed') {
|
||||
console.log(`✅ ${pkg}: PASSED`)
|
||||
if (result.output) console.log(` Output: ${result.output}`)
|
||||
} else if (result.status === 'skipped') {
|
||||
console.log(`⏭️ ${pkg}: SKIPPED`)
|
||||
} else if (result.status === 'error') {
|
||||
console.log(`❌ ${pkg}: ERROR`)
|
||||
if (result.error) console.log(` Error: ${result.error}`)
|
||||
if (result.stderr) console.log(` Stderr: ${result.stderr}`)
|
||||
}
|
||||
}
|
||||
|
||||
if (!testPassed) {
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Main execution
|
||||
const args = process.argv.slice(2)
|
||||
|
||||
if (args.length === 0) {
|
||||
// Run all tests
|
||||
await runUnifiedTests()
|
||||
} else if (args.length === 1) {
|
||||
// Run specific test across all packages
|
||||
await runSpecificTest(args[0])
|
||||
} else if (args.length === 2 && args[0] === '--lang') {
|
||||
// Run all tests for specific language
|
||||
await runUnifiedTests([args[1]])
|
||||
} else if (args.length === 2 && args[0] === '--langs') {
|
||||
// Run all tests for specific languages
|
||||
await runUnifiedTests(args[1].split(','))
|
||||
} else if (args.length === 3 && args[0] === '--lang') {
|
||||
// Run specific test for specific language
|
||||
await runSpecificTest(args[2], args[1])
|
||||
} else if (args.length === 3 && args[0] === '--langs') {
|
||||
// Run specific test for specific languages
|
||||
await runSpecificTest(args[2], args[1])
|
||||
} else {
|
||||
console.log('Usage:')
|
||||
console.log(' node test-runner.js # Run all tests')
|
||||
console.log(' node test-runner.js <test-name> # Run specific test on all packages')
|
||||
console.log(' node test-runner.js --lang <lang> # Run all tests for specific language')
|
||||
console.log(' node test-runner.js --langs <lang1,lang2> # Run all tests for specific languages')
|
||||
console.log(' node test-runner.js --lang <lang> <test-name> # Run specific test for specific language')
|
||||
console.log(' node test-runner.js --langs <lang1,lang2> <test-name> # Run specific test for specific languages')
|
||||
process.exit(1)
|
||||
}
|
||||
Reference in New Issue
Block a user