W3 test reorg · Pull Request #1891 · RDFLib/rdflib (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation18 Commits15 Checks0 Files changed

Conversation

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

Summary of changes

Migrated the various test_*_w3c.py test files into test/test_w3c_spec/
Removed the DAWG test suite because it's a legacy, un-updated version of the W3C sparql11 test suite (the latter not currently included as an RDFLib test suite)
Added the W3C sparql11 test suite to test/data/suites/w3c as a replacement for the old DAWG tests, renamed test_dawg.py as test_sparql_w3c.py and relocated as mentioned above.
Migrated the contained-within-DAWG RDFLib-specific SPARQL test suite (rdflib) to test/data/rdflib-sparql (with the intention of later breaking out the test separately from its current inclusion at the end of test/test_w3c_spec/test_sparql_w3c.py)
Changed the content of the (unused) test/data/suites/n3 (which isn't an element of the published W3C RDF test suite) to that of the W3C N3 test suite because it's an updated version of the same test suite, cloned test_turtle_w3c.py as test_n3_w3c.py, adjusted it to run and added the failures as a pro tem internal, marked-as-xfail "skiplist".
Replaced the elderly version of the Turtle test suite included in the W3C RDF test suite (as turtle) with the updated version of the same tests in the N3 test suite TurtleTests subfolder, fixing the incorrect reference in manifest.ttl as necessary to get it to work.
Pythonised the skiptests.list as test/skipped_sparql_tests.py to make it more tractable and adjusted test/test_w3c_spec/test_sparql_w3c.py to suit.
Added four (marked-as-xfail) unicode numeric codepoint escape tests to the RDFLib-specific SPARQL test suite test/data/rddlib-sparql
Added a couple of TurtleTest URIRefs to test/manifest.py and enabled the execution of DAWG-proposed tests, pro tem
Renamed the existing W3C RDF test suite rdfxml subdirectory as rdf-xml to update it to current W3C structure
Transcribed buried, commented-out tests in DAWG (e.g.

# :term-6 # Not SPARQL 1.1

) as entries in skiplist.

# :term-6 # Not SPARQL 1.1
) as entries in skiplist.

(Or, the above can be implemented as half-a-dozen separate PRs if desired)

Checklist

Checked that there aren't other open pull requests for the same change.
Added tests for any changes that have a runtime impact.
Checked that all tests and type checking passes.
Considered granting push permissions to the PR branch,
so maintainers can fix minor issues and keep your PR up to date.

Graham Higgins added 4 commits

May 7, 2022 16:03

Coverage remained the same at 88.248% when pulling 766ccc3 on gjhiggins:w3-test-reorg into b99d022 on RDFLib:master.

@ghost ghost self-requested a review

May 7, 2022 17:55

@gjhiggins mind if I split this up a bit just to make it easier to review?

Also, if I run the fetcher the following files change - not sure why, I think maybe because you are using the test suites from github, if so we should correct the URLs but maybe it is best to consider updating the test suites independently also.

$ git log -1 commit 20188357204f8c687654fc03545d63f95abed0df (HEAD -> w3-test-reorg, gjhiggins/w3-test-reorg) Author: Graham Higgins gjh@bel-epa.com Date: Sat May 7 18:19:52 2022 +0100

use pubiished SPARQL test suite data

$ git status On branch w3-test-reorg Your branch is up to date with 'gjhiggins/w3-test-reorg'.

nothing to commit, working tree clean $ task test:data:fetch task: [test:data:fetch] .venv/bin/python test/data/fetcher.py 2022-05-07T20:58:25.953 168177 139962706229056 020:INFO root fetcher:87:fetch fetching https://github.com/w3c/N3/archive/c44d123c5958ca04117e28ca3769e2c0820f72e6.zip to temp file /tmp/tmpcv22er67/i6qH988zpZ.zip 2022-05-07T20:58:27.326 168177 139962706229056 020:INFO root fetcher:122:fetch Downloaded https://github.com/w3c/N3/archive/c44d123c5958ca04117e28ca3769e2c0820f72e6.zip and extracted files matching re.compile('^[^\/]+[\/]tests\/$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/n3 2022-05-07T20:58:27.708 168177 139962706229056 020:INFO root fetcher:87:fetch fetching https://www.w3.org/2013/TurtleTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/LdC9B1dlIa.tar.gz 2022-05-07T20:58:27.930 168177 139962706229056 020:INFO root fetcher:122:fetch Downloaded https://www.w3.org/2013/TurtleTests/TESTS.tar.gz and extracted files matching re.compile('^[^\/]+\/$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/turtle 2022-05-07T20:58:28.298 168177 139962706229056 020:INFO root fetcher:87:fetch fetching https://www.w3.org/2013/N-QuadsTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/S1MSjxntZM.tar.gz 2022-05-07T20:58:28.332 168177 139962706229056 020:INFO root fetcher:122:fetch Downloaded https://www.w3.org/2013/N-QuadsTests/TESTS.tar.gz and extracted files matching re.compile('^(.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/nquads 2022-05-07T20:58:28.705 168177 139962706229056 020:INFO root fetcher:87:fetch fetching https://www.w3.org/2013/N-TriplesTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/JPjAizxL0A.tar.gz 2022-05-07T20:58:28.722 168177 139962706229056 020:INFO root fetcher:122:fetch Downloaded https://www.w3.org/2013/N-TriplesTests/TESTS.tar.gz and extracted files matching re.compile('^(.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/ntriples 2022-05-07T20:58:29.107 168177 139962706229056 020:INFO root fetcher:87:fetch fetching https://www.w3.org/2013/TrigTests/TESTS.tar.gz to temp file /tmp/tmpcv22er67/hQ2aGXyl1D.tar.gz 2022-05-07T20:58:29.333 168177 139962706229056 020:INFO root fetcher:122:fetch Downloaded https://www.w3.org/2013/TrigTests/TESTS.tar.gz and extracted files matching re.compile('^(.+)$') to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/suites/w3c/trig 2022-05-07T20:58:29.575 168177 139962706229056 020:INFO root fetcher:52:fetch Downloaded http://www.w3.org/2000/01/rdf-schema to /home/iwana/sw/d/github.com/iafork/rdflib.cleanish/test/data/rdfs.ttl $ git status On branch w3-test-reorg Your branch is up to date with 'gjhiggins/w3-test-reorg'.

Changes not staged for commit: (use "git add/rm ..." to update what will be committed) (use "git restore ..." to discard changes in working directory) modified: test/data/suites/w3c/n3/manifest.ttl deleted: test/data/suites/w3c/turtle/IRI-resolution-01.nt deleted: test/data/suites/w3c/turtle/IRI-resolution-01.ttl deleted: test/data/suites/w3c/turtle/IRI-resolution-02.nt deleted: test/data/suites/w3c/turtle/IRI-resolution-02.ttl deleted: test/data/suites/w3c/turtle/IRI-resolution-07.nt deleted: test/data/suites/w3c/turtle/IRI-resolution-07.ttl deleted: test/data/suites/w3c/turtle/IRI-resolution-08.nt deleted: test/data/suites/w3c/turtle/IRI-resolution-08.ttl modified: test/data/suites/w3c/turtle/README deleted: test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList.nt deleted: test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList.ttl deleted: test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList_of_two_objects.nt deleted: test/data/suites/w3c/turtle/blankNodePropertyList_as_object_containing_objectList_of_two_objects.ttl deleted: test/data/suites/w3c/turtle/localName_with_PN_CHARS_BASE_character_boundaries.nt deleted: test/data/suites/w3c/turtle/localName_with_PN_CHARS_BASE_character_boundaries.ttl modified: test/data/suites/w3c/turtle/localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries.nt modified: test/data/suites/w3c/turtle/localName_with_assigned_nfc_PN_CHARS_BASE_character_boundaries.ttl modified: test/data/suites/w3c/turtle/manifest.ttl deleted: test/data/suites/w3c/turtle/predicateObjectList_with_blankNodePropertyList_as_object.nt deleted: test/data/suites/w3c/turtle/predicateObjectList_with_blankNodePropertyList_as_object.ttl deleted: test/data/suites/w3c/turtle/test-38.nt deleted: test/data/suites/w3c/turtle/test-38.ttl deleted: test/data/suites/w3c/turtle/turtle-eval-lists-01.nt deleted: test/data/suites/w3c/turtle/turtle-eval-lists-01.ttl deleted: test/data/suites/w3c/turtle/turtle-eval-lists-02.nt deleted: test/data/suites/w3c/turtle/turtle-eval-lists-02.ttl deleted: test/data/suites/w3c/turtle/turtle-eval-lists-03.nt deleted: test/data/suites/w3c/turtle/turtle-eval-lists-03.ttl deleted: test/data/suites/w3c/turtle/turtle-eval-lists-04.nt deleted: test/data/suites/w3c/turtle/turtle-eval-lists-04.ttl deleted: test/data/suites/w3c/turtle/turtle-eval-lists-05.nt deleted: test/data/suites/w3c/turtle/turtle-eval-lists-05.ttl modified: test/data/suites/w3c/turtle/turtle-subm-01.nt modified: test/data/suites/w3c/turtle/turtle-subm-27.nt modified: test/data/suites/w3c/turtle/turtle-subm-27.ttl modified: test/data/suites/w3c/turtle/turtle-syntax-bad-n3-extras-06.ttl deleted: test/data/suites/w3c/turtle/turtle-syntax-blank-label.nt deleted: test/data/suites/w3c/turtle/turtle-syntax-ln-colons.nt deleted: test/data/suites/w3c/turtle/turtle-syntax-ln-dots.nt deleted: test/data/suites/w3c/turtle/turtle-syntax-pname-dots.ttl

Untracked files: (use "git add ..." to include in what will be committed) test/data/suites/w3c/turtle/turtle-syntax-lists-01.ttl test/data/suites/w3c/turtle/turtle-syntax-lists-02.ttl test/data/suites/w3c/turtle/turtle-syntax-lists-03.ttl test/data/suites/w3c/turtle/turtle-syntax-lists-04.ttl test/data/suites/w3c/turtle/turtle-syntax-lists-05.ttl

no changes added to commit (use "git add" and/or "git commit -a")

Also, if I run the fetcher the following files change - not sure why, I think maybe because you are using the test suites from github, if so we should correct the URLs but maybe it is best to consider updating the test suites independently also.

I'm guessing it is because of:

Replaced the elderly version of the Turtle test suite included in the W3C RDF test suite (as turtle) with the updated version of the same tests in the N3 test suite TurtleTests subfolder, fixing the incorrect reference in manifest.ttl as necessary to get it to work.

I would rather not do that, if w3c wants to release a new test suite it should be published here IMO: https://www.w3.org/TR/rdf11-testcases/ - and if we have to modify tests then the test suite is anyway problematic.

Hack it about as you will, I'll follow closely and review promptly.

The turtle directory in W3C's RDF test suite is 4 years old and is a frozen dupe of TurtleTests in the W3C N3 suite. Otherwise, I was aiming for replicating a checkout of the W3C RDF suite and just ignoring the turtle subdirectory in favour of the current one in the N3 test suite.

The "issue" is confined to manifest files, https://github.com/w3c/N3/blob/c44d123c5958ca04117e28ca3769e2c0820f72e6/tests/manifest.ttl#L13 is incorrect, there is no manifest,ttl in N3Tests so the RDFLib test suite execution produces an Exception --- however, there is a manifest-syntax.ttl present, so that's what I updated it to use. (I would imagine that Gregg Kellogg would appreciate a PR).

The turtle directory in W3C's RDF test suite is 4 years old and is a frozen dupe of TurtleTests in the W3C N3 suite.

But it the official test suite, it is even the same as in https://github.com/w3c/rdf-tests/tree/main/ntriples I believe. It may be frozen but so is the spec, and without some official statement accompanying it's release I don't think we should change it. More tests may be fine but if we want to report compliance with the ntriples test suite the suite we should be reporting compliance to is the official test suite.

But it the official test suite

I don't have any kind of agenda here, I just discovered that test/data/suites/w3c/n3 was completely unused by RDFLib tests and it isn't part of the current W3C RDF test suite so I went looking for an "official" W3C N3 test suite and found https://github.com/w3c/N3/tree/master/tests and recruited the NTriples tests.

it is even the same as in https://github.com/w3c/rdf-tests/tree/main/ntriples I believe.

Apart from the turtle tests, the W3C N3 test suite contents don't seem to have any other correlates in the W3C RDF test suite.

It may be frozen but so is the spec, and without some official statement accompanying it's release I don't think we should change it.

Sure, I have no problem with that stance.

I suspect the difference between the fetcher results and the current RDFLib w3c content is due to the fact that the RDFLib w3c hasn't been updated since 2015.

Just to summarise:

Move test/data/suites/DAWG/rdflib up to test/data/suites/rdflib-sparql and adjust the reference in test_dawg.py accordingly.
Delete test/data/suites/DAWG/
Pull sparql11 into test/data/suites/w3c, adjust reference in test_dawg.py
Rename test/data/suites/w3c/rdfxml to test/data/suites/w3c/rdf-xml, in prep for refurbishing RDFLib's copy of the W3C RDF test suite, adjust test/data/suites/DAWG/data-r2 and test/data/suites/DAWG/data-sparql11 references in test_dawg.py accordingly.
Ohhh, _ now I see ... I hadn't grokked that you'd only just added test/data/suites/w3c/n3 --- 'cos I couldn't find anything that actually tested it, I had it confused with the old n3 directory (which was renamed to n3roundtrip) and that's why I made the error of replacing your d/l of the W3C N3 test suite with my d/l of the same files. Anyway, you might want to keep the n3 test I added as test_n3_w3c.py, although the main test manifest is still borked as discussed.

Will start splitting off things from here either today or later in W19.

And yes, the n3 test suite data was only added recently in #1860 because I'm using it in #1858 (which still needs 1 review) and I did not want to make the fix PR too big to keep it easy to review.

Will merge and resolve tomorrow - also hoping for this to merge w3c/N3#99.

Will merge and resolve tomorrow - also hoping for this to merge w3c/N3#99.

Worth waiting for, Gregg Kellogg's clearly on the case.

Still busy slicing this up, in this PR you moved test/data/suites/DAWG/data-r2 to test/data/suites/w3c/sparql11/data-r2 - data-r2 is however the test suite for SPARQL 1.0, so I don't think it should be under sparql11, I'm going to instead rather use the name test/data/suites/w3c/dawg-data-r2

This was referenced

May 11, 2022

@gjhiggins I split off some more PRs but there is still some work left here, will likely only finish this next week as I have other priorities also.

This is now in test/data/suites/w3c/data-r2/:

#1908

We should be testing with unaltered official test data, issues with the test data should be reported upstream and accomodations for issues should be made in code.

I restored the turtle test suite to match the upstream version now, basically what is left is:

Switch from test/data/suites/DAWG/data-r2 to test/data/suites/w3c/dawg-data-r2
Replace test/skiptests.list with test/skipped_sparql_tests.py
Move test/data/suites/DAWG/rdflib to test/data/rdflib-sparql/
... (some more stuff I don't want to write down now)

This was referenced

May 13, 2022

I will try finish this today still, these are all very useful changes @gjhiggins - so thanks you, hard to deal with them in one PR but still good they are done.