ci: run the Java integration test-suite on GitHub against AIO (jenkins offboarding D) by mo-auto · Pull Request #14225 · JanssenProject/jans (original) (raw)
added 2 commits
…one image (offboarding D1)
Add CN_PERSISTENCE_LOAD_TEST_DATA (default false) so the Janssen integration-test dataset can be loaded into the SQL backend, enabling the test-suite to run against the all-in-one image without jenkins.jans.io.
- test_data_setup.py mirrors jans-linux-setup test_data_loader for SQL: adds the custom test columns (parsed from the vendored test schema LDIFs), imports the auth/scim/fido2 test data LDIFs, adds the password grant to the SCIM client, applies the jans-auth dynamic-config delta, enables the test scripts and promotes the test default scopes. Idempotent and lock-guarded.
- bootstrap.py runs it as a gated stage after the custom-ldif stage.
- vendor templates/test into the persistence-loader image.
- all-in-one: default CN_PERSISTENCE_LOAD_TEST_DATA=false.
- config-api: honor CN_CONFIG_API_TEST_CLIENT_ID/SECRET/TRUSTED so the config-api test client is deterministic for the integration test-suite.
The config-api test client already receives every scope on upgrade (update_test_client_scopes), so no config-api test data is loaded here.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…fboarding D2)
Add .github/workflows/test-integration.yml + scripts/render_test_profiles.py to run the HTTP integration test-suites against an all-in-one server with test data loaded, on a MySQL/PGSQL matrix - replacing the jenkins.jans.io / docker-jans-monolith runner.
- builds persistence-loader + config-api from the checkout and assembles the AIO so the test-data changes are exercised; runs the DB + AIO + a TLS-terminating proxy on 443
- extracts the live salt / SCIM / config-api secrets from the AIO through its own pycloudlib manager (adapter-agnostic) and renders profiles// for each module from the canonical templates/test sources; copies the committed client keystores
- imports the AIO cert into the JDK truststore, builds the modules, then runs 'mvn -Dcfg= test' for jans-auth/client, jans-scim/client, jans-config-api and jans-fido2/client (continue-on-error, gated at the end)
- publishes results three ways: GITHUB_STEP_SUMMARY table, uploaded artifact, and a SHA-pinned dorny/test-reporter PR check; dumps AIO logs on failure
- all actions SHA-pinned + harden-runner; matrix is MYSQL-only on PRs, both nightly
Server-side persistence-coupled suites (jans-auth-server/server, jans-orm) and agama are a follow-up.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…l+vault+traefik)
The AIO requires consul + vault (pycloudlib defaults CN_CONFIG_ADAPTER=consul / CN_SECRET_ADAPTER=vault); the previous hand-rolled 'docker run' started neither, so the configurator hung and the AIO never became healthy (first run #27060489146).
Reuse the canonical automation/start_janssen_aio_demo.sh which brings up the full compose stack (consul + vault + traefik TLS + db + AIO). Bake the integration-test env (CN_PERSISTENCE_LOAD_TEST_DATA + CN_CONFIG_API_TEST_CLIENT_*) into a thin image layer tagged as the image the demo compose expects, so the script is used unmodified. Import the demo's generated CA cert into the JDK truststore; extract live secrets from the 'jans' container.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…er configurator
start_janssen_aio_demo.sh generates the TLS certs as mode-600 files owned by the host user; the AIO configurator runs as uid 1000 and got PermissionError reading the mounted ca.key, so configurator-load failed and the auth_jks_base64 secret was never created (AIO stayed unhealthy). Relax the cert perms and restart the AIO after the stack is up.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…data loader
pycloudlib's create_from_ldif only inserts plain entries, but the test data LDIFs also carry 'changetype: modify' records (flip jansDefScope on extra scopes, enable extra scripts, activate an attribute, add group members, add scope claims). Those records have no objectClass, so _data_from_ldif crashed with 'NoneType object is not subscriptable' (oc[-1] on None).
Split each LDIF into plain entries (-> create_from_ldif) and modify records, and apply the modifies via client.update with a DN-suffix->table map, inferring scalar/multivalued + boolean columns from the existing value. Comment-only/no-DN blocks are skipped.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…a reflect + resilience)
The modify applier crashed (AttributeError: NoneType has no attribute 'c') when client.get hit a table missing from the cached SQLAlchemy metadata. Force a fresh metadata reflection before applying modifies, skip a modify whose table still isn't reflected, wrap each modify so one failure can't abort the whole load, and initialise multivalued 'add' targets with the correct JSON shape when the column is empty.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…oad best-effort
Some test entries carry attributes (e.g. creationDate on jansFido2RegistrationEntry / jansCustomPerson) that are not columns of their SQL table, so pycloudlib's insert raised 'CompileError: Unconsumed column names'. Pass a transform_column_mapping callback to create_from_ldif that drops attributes without a matching reflected column. Also wrap the post-import config tweaks so a single failure can't crash-loop the loader and keep the AIO unhealthy.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
… failure
The AIO comes up (loader completes, fido2 deploys) but jans-auth isn't serving openid-configuration (404) and the tail-400 dump truncates the service logs. Capture supervisorctl program states + a targeted jans-auth/error grep so the jans-auth startup failure is diagnosable.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
- fix(casa): correct otp-java import package to com.bastiaanjansen.otp
otp-java's Maven groupId is com.github.bastiaanjansen, but its Java package is com.bastiaanjansen.otp. The OTP migration used the groupId as the import package, so casa failed to compile ("package com.github.bastiaanjansen.otp does not exist"). Fix the imports in the casa HOTP/TOTP services and the two OTP person-auth scripts. The pom dependency groupId (com.github.bastiaanjansen:otp-java) is correct and unchanged.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
- fix(casa): pass counter to HOTP getURI (otp-java API)
otp-java's HOTPGenerator.getURI takes the moving-factor counter first: getURI(int counter, String issuer, String account). Pass 0 (initial counter, matching the previous lochbridge default) in the casa HOTP service and the HOTP URI calls in both OTP scripts. TOTPGenerator's getURI(issuer, account) is correct and unchanged. Verified all other otp-java calls (Builder, generate, verify, HMACAlgorithm) against the v2.1.0 source.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com Co-authored-by: moauto 54212639+mo-auto@users.noreply.github.com
docker-jans-auth-server pulls casa-config + jans-fido2-client + jans-fido2-model + agama-inbound as custom libs from the release, but build-test only collected the first three, so agama-inbound-0.0.0-nightly.jar was 404 and the 'docker (auth-server)' image build failed (run 27098512184) - leaving a stale ghcr auth-server image (and breaking the AIO jans-auth).
agama-inbound is a jans-auth-server submodule (jans-auth-server/agama/inboundID, artifactId agama-inbound), installed to ~/.m2 during the auth-server deploy; add it to the release-asset collection alongside the other auth-server custom libs.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…ogress
Cut the health wait 1200s->600s so a stuck AIO fails fast, dump jans-auth/configurator log lines periodically during the wait, and enrich the failure dump (supervisorctl status + grepped service/error logs) so the reason jans-auth stays at 404 is visible without waiting/cancelling.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…ebKeys date
On AIO-health failure, dump the configurator auth-keys.json, the decoded auth_openid_key_base64 secret, and the jansConfWebKeys DB value, to pinpoint which stage turns the JWKS into a date string.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…tore corruption
The configurator's first run failed to read the demo's mode-600 TLS certs, and the restart workaround re-ran key generation against a half-initialised keystore -- writing a 'Keystore was tampered with' error string into jansConfWebKeys and breaking jans-auth and config-api WebKeysConfiguration parsing. Run the demo in the background and relax the cert perms as they appear so the configurator succeeds on its first run, no restart needed.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
… as root
The setup-java JDK's cacerts is not writable by the runner user (keytool failed with FileNotFoundException: .../cacerts Permission denied), so run the import via sudo using the explicit keytool path.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…e live FQDN
The build runs clean install over the full jans-auth-server/scim/config-api/fido2 reactors; -Dcfg= required a rendered profile for every filtering module (agama-engine has none -> 'Error loading property file .../agama/engine/profiles//config-agama-test.properties'). Compilation does not need the live FQDN, and every module ships a default profile, so build with -Dcfg=default and keep -Dcfg= for the test step (whose 4 legs are fully rendered).
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…run unit suites
Triage support: the failure log dump never ran (job only fails at the gate, so failure() was false), so dump jetty service logs (scim/config-api/auth/fido2) on always() to expose the 503/401 cause. Collect surefire reports workspace-wide (the jans-auth-client reports were not captured). Add a unit-suite step (jans-orm, jans-core).
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…ead of 503
jans-scim returns 503 (AuthorizationProcessingFilter.disabledApiResponse) on every CRUD call because jansScimEnabled is false: the persistence-loader defaults CN_SCIM_ENABLED=false and the workflow never set it (linux-setup always enables SCIM). Bake CN_SCIM_ENABLED=true into the AIO.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
…er calls work
The AIO demo pinned the FQDN to the host IP via the jans container's extra_hosts; with a loopback IP (CI) the in-container public URL had no :443 listener, so config-api's jans-auth introspection callback got 'Connection refused' and every config-api request returned 401. Alias the FQDN to the traefik service on the compose network (and drop the extra_hosts override) so https:// resolves to traefik's TLS entrypoint and routes back to the AIO.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
Mirror the Jenkins full-build-with-tests model: instead of building with -DskipTests and running only 4 client legs, run 'mvn clean install' with tests ENABLED and -Dcfg= over every reactor (jans-orm, jans-core, jans-auth-server, jans-scim, jans-config-api, jans-fido2), so each module's unit AND integration suites execute against the live AIO in one pass. Render the agama-engine profile so that reactor compiles under -Dcfg, and gate the job on the collected testng failure count (failure.ignore lets all suites run first).
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
[![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4)
- summarize_testng.py: render fully-qualified class names in the summary table + gate offenders (unambiguous triage; matches the FQN baseline).
- reap: match the exact SSH-key prefix jans-ci-- (trailing hyphen) so cleanup can't delete another run whose id shares the numeric prefix.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
[
iromli previously approved these changes Jun 12, 2026
…lient_trusted
- reap: page through all DigitalOcean SSH keys (follow links.pages.next) so the fallback cleanup isn't limited to the first 200 keys.
- config-api bootstrap: simplify test_client_trusted to str(as_boolean(...)).lower() (per review).
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
[
Collect matching key IDs across all pages first, then delete, so deletions can't shift the page window mid-walk and skip a key.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
[
moabu pushed a commit that referenced this pull request
The docker-jans-monolith image (all jans services in one container, for testing/dev) is superseded by the all-in-one (AIO) demo + the GitHub integration-test workflow (Phase D, #14225). Remove it and every reference:
- build-docker-image.yml: drop monolith from the matrix, ALL_SERVICES, the dispatch input, and the per-service selection.
- dependabot.yml: drop the /docker-jans-monolith entry.
- github-labels: drop the comp-docker-jans-monolith label.
- generate-rdbms-docs.sh: run against the AIO demo container (jans) instead of the monolith.
- docs: repoint the Docker quickstart to start_janssen_aio_demo.sh, delete the obsolete monolith compose.md, drop the jans-monolith row from the k8s image list.
Historical CHANGELOG entries are left intact.
Signed-off-by: moauto 54212639+mo-auto@users.noreply.github.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
[ Show hidden characters]({{ revealButtonHref }})