T187844 Support the glTF 2.0 3D file format (original) (raw)
Support the glTF 2.0 3D file format
- Edit Task
- Edit Related Tasks...
- Create Subtask
- Edit Parent Tasks
- Edit Subtasks
- Merge Duplicates In
- Close As Duplicate
- Edit Related Objects...
- Edit Commits
- Edit Mocks
- Mute Notifications
- Protect as security issue
This royalty-free standard format was enabled today on Facebook, which means it's likely to take off a bit (and can be exported from Blender).
The format is described at https://www.khronos.org/gltf/ supports textures and animations (which STL doesn't) and is designed to be efficient for transmission and client-side interpretation, which would work nicely for us.
Obviously this'd be a fair bit of work, not least the security concerns. Would be interesting to see what Facebook have done in this regard, and whether we could reuse their work if it's FLOSS.
It looks like most of what we need is in three.js, which we already use for the stl rendering, so rough plan for using the .glb binary variant would be:
- add .glb and proper mime type to core's mime.info
- add a validation hook for uploaded files to 3D (magic number, proper sections, valid json, no external uri references)
- split ThreeDHandler into a base class, with stl and glb subclasses to do validation and extraction differently for each supports type
- extract asset copyright and generator fields from the json as metadata when present, expose them in the table on File: page
- include three.js's gltf loader in 3D and 3d2png's js modules
- change thumbor config to pass glb through the 3d2png handler
- add mmv plugin config for glb as well as stl
- change varnish config to gzip the glb files on transfer
Related Changes in Gerrit:
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Note glTF is supported by the BabylonJS engine - https://doc.babylonjs.com/how_to/gltf - which is under Apache 2 license. I don't know how heavyweight it is for a minimal viewer (it's a fairly full-featured 3d engine which can be used for games and such as well as simple model viewing).
Peeking at the spec, the problem is going to be with external files for textures and binary data -- JSON glTF files can either contain their own data as base-64 data: URIs, or reference external files, so a glTF file *can* be standalone but doesn't *have* to be.
They'll need to be validated on upload to make sure no absolute URIs are used (evil!) or relative URI references left dangling (won't work).
There are also binary glTF files, which is a sort of archive format that bundles the JSON file along with a binary data chunk containing images/etc. It has a clear magic number which should make detection of file type easy, but the ordering of the data inside the binary chunk seems to be up to the glTF JSON description. (As with other binary file types it would be possible to embed Files We Don't Like semi-invisibly, such as embedding copyrighted images or sounds or movies. How much this is an issue I can't say.)
One more thing -- in the binary format, the raw binary chunk comes after the JSON chunk and has no footer/trailer at the end, so it'd be possible to create a binary glTF file that's parseable as .zip/.jar which uses a trailer instead of a header. Our existing upload-time checks should catch this case.
They'll need to be validated on upload to make sure no absolute URIs are used (evil!) or relative URI references left dangling (won't work).
Yeah, not too hard. Theoretically we could transform-on-upload with absolute URIs and re-package, but that feels icky.
Why not have the upload UI ask for referenced files (if any) and always roll it all into a single document, since that's supported?
Hi, one of the three.js contributors (and disclaimer, glTF working group member) here.
Thought I'd point out that Facebook is requiring all uploads use the binary (.glb) version of glTF — they don't accept any external files whatsoever. It would be reasonable to make the same requirement here. The Blender exporter has a feature to embed everything to binary, and options exist for packing existing glTF files into a single .glb document as well. Let me know if I can help with anything. :)
Nice, that should simplify things if we only accept binary .glb. :) As far as I can tell from spec a binary file can still reference external resources if it wants, but the validation step could detect that easily and return an explanatory error message.
Oh -- and as a todo note we'll want to add http-level gzip compression for the .glb files too, just as we added for .stl. Easy to do and should save bandwidth on the JSON part of the blob and the meshes. (Textures are pre compressed, but gzip won't hurt on them.)
(I like the idea of letting you bundle multiple files together and do the transform in the UI, I just expect it to be tricky to integrate into UploadWizard. Will be faster to get going if we leave that to external tooling.)
As far as I can tell from spec a binary file can still reference external resources if it wants...
That's correct; it's atypical but certainly possible. There's an official validator as well, which reports external resources along with many other issues, e.g. —
resources: [ {pointer: "/buffers/0", mimeType: "application/gltf-buffer", storage: "external", uri: "scene.bin", byteLength: 2617976} {pointer: "/images/0", mimeType: "image/png", storage: "external", uri: "textures/lainmat_baseColor.png", image: {…}} {pointer: "/images/1", mimeType: "image/png", storage: "external", uri: "textures/lainmatAO_baseColor.png", image: {…}} {pointer: "/images/2", mimeType: "image/png", storage: "external", uri: "textures/lainmatshadeless_baseColor.png", image: {…}} ]
It might be best to reject uploads with any outright errors, as we can't guarantee those will work in future versions of three.js.
Did a quick peek at the GLTFLoader for three.js. Couple things would need to be changed upstream to work in Node for 3d2png:
- it tries to use window.TextDecoder if available to decode the UTF-8 JSON blob; there's a fallback but it throws a runtime error before it gets to the fallback because window is undefined.
- it uses the Blob constructor and URL.createObjectURL to create web-loadable URIs for textures out of the buffer views. There is no Blob constructor, and no URL, in the node environment. Need to change it to use some other kind of loader?
- even if you had blob URLs, three's ImageLoader uses DOM stuff which won't work. Appears to be a pluggable system by regex, so easy to add support for a fake protocol like 'blob:'
Provisional patch to 3d2png doesn't render successfully yet, but does load the scene data. ;)
Now renders the "Avocado.glb" and "Boombox.glb" sample files successfully. :)
Had to adjust the near frustum on the camera from 1 to 0.01 since glTF files often seem to use meters for units, and our centering was putting us at a pretty close position within 1 meter. Also made a tweak to the node texture loading hack which makes textures actually work (use THREE.Texture, not THREE.DataTexture even though it sounds like what I want it doesn't work).
Shouldn't be super difficult to use the GLTFLoader in the web view once that frustum tweak is in (it won't need the rest of the node-specific loader tweaks, just a slight adjustment to centering to handle passing through Scene and Object3D objects to find the mesh).
Looking great! A few notes that may be helpful:
From spec, on coordinate systems:
- glTF uses a right-handed coordinate system, that is, the cross product of +X and +Y yields +Z. glTF defines +Y as up. The front of a glTF asset faces +Z.
- The units for all linear distances are meters.
And, although I haven't tested this in Node.js, CommonJS imports _should_ let you pull the specific versions associated with your three.js version this way:
const THREE = require('three'); require('three/examples/js/loaders/STLLoader'); require('three/examples/js/loaders/GLTFLoader');
var loader = new THREE.GLTFLoader();
This won't work with ES6 modules unfortunately. The community-maintained npm packages for individual loaders tend to be pretty out of date.
Hey I implemented gltf support in facebook so happy to jump in here however i can. I would definitely advise to start with a limited format set, ie just use glb, trying to convert everyone elses models into what you can understand is a daunting proposition.
You'll also need to figure out what extensions you want to support. At fb we support the KHR_materials_unlit extension as a lot of photogrammetry models use it, or models that want to bake really fancy lighting into the textures. Sketchfab seems to have a lot of SpecularGlossiness models which we try to convert to regular PBR but i wouldn't recommend it, at least to start.
Hi @Msfeldstein
Thanks for joining the conversation. I think we'll have a lot more questions for you soon :)
Hey I implemented gltf support in facebook so happy to jump in here however i can. I would definitely advise to start with a limited format set, ie just use glb, trying to convert everyone elses models into what you can understand is a daunting proposition.
You'll also need to figure out what extensions you want to support. At fb we support the KHR_materials_unlit extension as a lot of photogrammetry models use it, or models that want to bake really fancy lighting into the textures. Sketchfab seems to have a lot of SpecularGlossiness models which we try to convert to regular PBR but i wouldn't recommend it, at least to start.
@Deltakosh babylonjs is something to consider too! One thing is we need a headless mode to render flat thumbnails as well as the live viewer... We've managed to get three.js running under node with headless WebGL, though sometimes it's a little funky to add new features to it. Is there a headless mode available for babylon as well? (In theory we could change from node to headless Chrome or Firefox but that's a whole other thing to arrange with ops, probably. :)
It is also working and we have several users already doing it.
We went to Firefox on our side for ops (running visual tests for incoming PR)
Feel free to ping me directly if you need help: Davca@microsoft.com
This is really great additional 3D model support is being worked on. Can I ask if there is much documentation around converting popular 3D file formats like .OBJ and .PLY to this format? I'd hate for people to not use this due to lack of good quality documentation for easy process to convert the files to this format (like what happened with audio and video before .webm).
This is really great additional 3D model support is being worked on. Can I ask if there is much documentation around converting popular 3D file formats like .OBJ and .PLY to this format? I'd hate for people to not use this due to lack of good quality documentation for easy process to convert the files to this format (like what happened with audio and video before .webm).
Easy conversion & transfer was in the scope of Wikimedia Sverige’s 3D2Commons project − unsure of the current status but maybe worth checking out with them?
Short version: +1 for glTF
Long vversion:
glTF is the (currently) only format the Sketchfab Download API can output
On Sketchfab there are about 100 000 CC-licensed models (see https://twitter.com/albn/status/986267640883707909 ). Some are likely NC and so out of scope for Commons. I know the cultural heritage content from GLAMs the best and certainly there's a lot of good content for it.
The tool JeanFred mentions that the Swedish National Heritage Board (where I work) and Wikimedia Sweden were aiming to develop had as its first idea to basically be for Sketchfab what Flickr2Commons is for Flickr. But when we started the project the Sketchfab Download API was not released so we changed direction. If glTF support is coming to Commons the original idea would be in play and the National Heritage Board could entirely fund or co-fund development, (depends on total cost of course) of such a Sketchfab2Commons style tool.
May I reanimate this request? Gltf/GLB have become a kind of standard for 3D models in the last years. WikiMedia supports STL-Files for a while now. But STL is only for 3D-Printing and includes no colours, no textures. The viewer https://www.mediawiki.org/wiki/Extension:3DAlloy is based on ThreeJS and so could also render GLB.
I see at last two main use cases:
- Showing models in WikiPedia pages. I would be great, to see the Cydney Opera or a Police Box(Tardis) textured.
- Placing the models on a map/OpenStreetMap. A reference in the OSM tagging enables map viewer to load and show the Opera or the Tardis at the right places.
This got brought up recently and I think it's a reasonably self-contained and not huge work blob. :) Adding myself to do a spike test bringing branch up to date and either making it work or deciding we should redo it differently with more resources.
Thank you @bvibber!
no lighting? needs fixing obviously
I don't think I can comment on the changeset, so chiming in here. three.js made some significant changes to the lighting system in r155. There's a migration guide for lighting. That said, if you're after good general-purpose lighting for unknown input files, I would strongly recommend THREE.RoomEnvironment, as shown in this example.
RoomEnvironment is relatively foolproof to set up, and will work much better for physically-based rendering (PBR) materials — as frequently found in the glTF format — than point/spot/directional/ambient lighting. It's a very lightweight implementation of image-based lighting (IBL) and you'll find it used elsewhere in viewers like https://modelviewer.dev/ and https://gltf.report/.
RoomEnvironment is relatively foolproof to set up, and will work much better for physically-based rendering (PBR) materials — as frequently found in the glTF format — than point/spot/directional/ambient lighting.
eeeeexcellent, that'll be the target of my hacking tonight & tomorrow to a) restore lighting and b) get gtlf working :D
Sketchfab has announced it will be closing/migrating to Epic's new 'Fab' marketplace by 2025..
Mid-October
Fab will launch! The Sketchfab Store will close; models that had previously been available for purchase will still be viewable on Sketchfab but will no longer be available for purchase there. If you have purchased content from the >Sketchfab Store, you will continue to have access to it through the Purchases tab of your Sketchfab profile page.If you migrated CC-BY or Standard licensable models to Fab, those Sketchfab model pages will remain available to view on Sketchfab but will offer a link out to Fab for users who might wish to download or purchase your models. >Models that were available on Sketchfab under other license types, e.g., Editorial or CC-BY-SA, will remain available to view on Sketchfab but will not provide a link to Fab on the model page.
Future roadmap
In 2025, we plan to stop offering downloadable content on Sketchfab. All creators who have not already moved their CC-BY licensable models to Fab will be invited to do so in advance of this change. We plan to ensure that >existing integrations that use Sketchfab’s Download API to access the free content library will continue to work until new APIs are made available on Fab. Also in 2025, Sketchfab’s enterprise solutions will transition to Fab.
it'd be good to have gltf support in place on commons by the time this happens.
Hi @bvibber, are you currently working on the extension? Otherwise I would give it a try. It would be great if you could take a look at my comment on 1026883.
I have created a prototype for a viewer that also supports GLB files in the ‘mediawiki’ style. Only simple GLB files and KTX2 compression are supported. I am not yet satisfied with the positioning of the models, especially in view of the thumbnails. Help and feedback are very welcome.
Great! I would be happy if the mediawiki enables GLB to.
My test today:
https://github.com/OpenDEM/WikiMediaExtension3D_Test_GLB_Format/issues/2
As I have seen this was already looked promising in tested. What is a roadblock or any issue that this feature was not include in mediawiki (and supported in the whole wiki project) as of of yet?
I would recommend to the uploader to use the original scale as international unit: 1 should be 1 meter/metre
Looking great! A few notes that may be helpful:
From spec, on coordinate systems:
- glTF uses a right-handed coordinate system, that is, the cross product of +X and +Y yields +Z. glTF defines +Y as up. The front of a glTF asset faces +Z.
- The units for all linear distances are meters.
And, although I haven't tested this in Node.js, CommonJS imports _should_ let you pull the specific versions associated with your three.js version this way:
const THREE = require('three'); require('three/examples/js/loaders/STLLoader'); require('three/examples/js/loaders/GLTFLoader');
var loader = new THREE.GLTFLoader();
This won't work with ES6 modules unfortunately. The community-maintained npm packages for individual loaders tend to be pretty out of date.
Another hello here...I do a lot of work with the Khronos group (keepers of glTF) and can hopefully help. We view the ability to natively display 3D (via glTF) as quite important and a big win! I'm technically clueless, (not totally) but can put the right people together ...and I've been doing this web 3D stuff for ages. New to phabricator as well so be kind..feel free to ping me.
sent you an email request for the video link...thanks in advance
I send you an email at 11:59, and now again. If the mail has not arrived please let me know.
I got the email..but I don't see any link to the jitsi call?
thanks..Sandy
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits