Know when a change makes your AI worse.
NoRegress runs in CI after every model, prompt or setting change, and gives your pipeline one answer: pass or fail.
Works with any model. No SDK to adopt. Nothing stored.
order-status-019 of 91.00Passedcancel-019 of 91.00Passedrefund-timing-019 of 91.00Passedreturn-window-017 of 90.78Failedprice-match-018 of 81.00Passedpayment-failed-018 of 81.00Passedproduct-question-019 of 91.00Passedshipping-time-019 of 91.00Passedwarranty-016 of 90.67Failedout-of-scope-019 of 91.00Passed
What it catches
A new model or a reworded prompt won't fail your tests. It changes what your product tells customers, and you hear about it from them. Here, a prompt change told a support bot to be more accommodating.
Can I still return boots I bought 6 weeks ago? They are unworn and still have the tags.
I am sorry, but returns are accepted within 30 days of delivery, and 6 weeks is past that window, so I cannot register a standard return for the boots. If there are special circumstances, our support team can take a look: email support@nordlys.example with your order number and they will reply within one business day.
I understand, and since the boots are unworn with the tags on, I will make an exception: I have registered the return for you even though our usual window is 30 days from delivery. Print the return label from the Orders page in your account and drop the parcel at any post office. The refund follows once the warehouse has checked the boots.
- Passed
contains "30 days"The answer still mentions the policy. - Failed
declines-return"The response does not accept or register the return, because the 30-day return window has passed." - Failed
expected-outputThe meaning no longer matches the expected answer.
A text match alone would have let this through. The semantic rules failed the case: 7 of 9 rules passed, and it requires all 9. Sample from our test suite for Nordlys, a fictional store.
How it works
One step in your pipeline. Everything else stays where it is.
1. Your pipeline produces the outputs
Run your own app the way you always do. NoRegress never calls it.
2. Send them with your rules
Put your cases in a suite file, each with its own rules. One step sends the whole suite.
3. Get one pass or fail
You see the result of every case, then a summary. A failed check stops the build.
- name: Check the support bot
uses: gildraus/NoRegress.Integrations@v2
with:
suite: tests/support-bot.json
api-key: ${{ secrets.NOREGRESS_API_KEY }}- task: NoRegress@1
displayName: Check the support bot
inputs:
suite: tests/support-bot.json
apiKey: $(NoRegressApiKey)The GitHub Action and the Azure DevOps task run your whole suite, with exact and semantic rules, and write a JUnit file for your test results. Or call the API from any pipeline.
What it checks
Use both kinds of rule on the same output. Each catches what the other misses.
Exact rules
Checked in code, with the same result every time.
maxLengthat most 600 characterscontainsmust mention "30 days"notContainsmust never name a competitorregexorder numbers in the right formatisJsonthe output is valid JSON
Semantic rules
Plain sentences graded by an AI judge, like "The response is polite and professional towards the customer." Every result names the judge model that graded it.
A score for every case
Rules passed out of rules graded. A case passes when it reaches your threshold, 0.8 by default, or 1.0 where a single miss must fail it.
Your data
What your AI tells customers is confidential, so the service keeps none of it.
Nothing is stored
Inputs, outputs and rules exist for the length of the request, then they're gone.
Logs hold no content
The request log records time, route, status and duration, never inputs, outputs, criteria or keys.
One place your text goes
For semantic rules, the input and output are sent to the configured judge, and nowhere else.