Rejected: Storing raw VSK_map files via DVC to avoid limitations

Categories:

  • pipeline
  • V-Sekai

Status:

  • rejected

Deciders:

  • V-Sekai
  • ifire
  • lyuma

Context and Problem Statement

How do I version control large binary game assets?

Need a procedure to version control large file folders like game map levels.

Describe the proposed option and how it helps to overcome the problem or limitation

Propose using DVC as the version control system on top of git with a S3 remote on Digital Ocean.

Describe how your proposal will work, with code, pseudo-code, mock-ups, or diagrams

Initial list of binaries to transfer to DVC

Use a Linux command to get the top 20 by size file types.

  • *.glb
  • *.gltf
  • *.bin*
  • *.png
  • *.webp
  • *.jpg
  • *.jpeg
  • *.exr
  • *.scn

Remotes

S3 Compatible has multiple implementations. Intent on using digital ocean spaces.

Permissions vs Cost

  • Permissions?
  • Open bucket
  • Cost of storage

Installing

  • https://dvc.org/doc/install/windows
    • https://github.com/iterative/dvc/releases/download/2.9.5/dvc-2.9.5.exe
  • https://dvc.org/doc/install/linux

Test using chibifire dvc bucket and the avatar wellness repository.

git clone https://github.com/fire/avatar-wellness.git
dvc init
dvc remote add -d chibifire-dvc-web https://chibifire.sfo3.digitaloceanspaces.com/dvc/
# dvc remote add --local chibifire-dvc-s3 s3://chibifire/dvc
# dvc remote modify --local chibifire-dvc-s3 endpointurl sfo3.digitaloceanspaces.com
# dvc remote modify --local chibifire-dvc-s3 access_key_id 'AAAA'
# dvc remote modify --local chibifire-dvc-s3 secret_access_key 'BBBB'
dvc config core.autostage true

A V-Sekai alternative organization for creator

Discussed with Lyuma how it is possible to make a V-Sekai alternative organization for creator repositories.

Positive Consequences

  • Able to version control maps.

Negative Consequences

  • Usability now has a gap.

Option graveyard:

  • Option: Do not use DVC remote on Google drive.

  • Rejection Reason: Google Drive does not have two implementations

  • Option: Use Github with git-lfs

  • Rejection Reason: The last time I checked it was 50 USD per 50 gigabytes.

  • Option: Use Gitlab git-lfs

  • Rejection Reason: There is slightly larger limit on https://gitlab.com git-lfs but gitlab.com does not scale with usage.

  • Option: Use Gitlab.com

  • Rejection Reason: Gitlab.com commrcial is expensive

If this enhancement will not be used often, can it be worked around with a few lines of script?

A version control system and object stores are not trivial and hard.

Is there a reason why this should be core and done by us?

We want to present V-Sekai maps for others to remix.

References

Derivative License

Copyright (c) 2022 V-Sekai contributors.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.