3D Archiving Houses and Settlements
Alternative Technologies to support Lasting Diachronic Memory
KOBAYASHI, Hideyuki (DR. Eng.)*
Abstract
In order to preserve 3D digital data describing houses, we propose alternative technologies to
keep the readability of the data against ever changing data formats, and lasting recording media,
through usage of ‘virtual converter’, and ‘electronic record’ attached to each building, as
follows:
Background and purpose
1. Cases of organizations in charge of maintaining lasting data
2. Historical review of media recording digital data
3. Currently used data formats for modeling 3D objects
4. Development of alternative technologies
5. Next steps
Conclusion
Keywords: diachronic archiving, 3D data format, recording media, houses and settlements
*Research Coordinator for Housing Information System,
Research Center for Advanced Information Technology,
National Institute for Land and Infrastructure Management (NILIM),
Ministry of Land, Infrastructure, Transport and Tourism (MLIT), JAPAN
E-Mail: kobayashi-h92qa@nilim.go.jp
Background and purpose
In Japan, housing stock already far exceeds the total number of household, therefore,
maintenance of existing houses and buildings became more important, rather than massive
construction and provision.
(million units)
Total house unit (left index)
Vacant house units (left index)
Vacant rate (right index)
1958 1963 1968 1973 1978 1983 1988 1993
1998 2003 2008
Figure-1 Trend of total house units, vacant units and vacant rate in Japan 1958-20081)
According to the national census on housing and land 2008, 49.99 million households were
counted, while total housing stock was 57.59 million units. 7.56million units were identified
as ‘empty house’ (fig 1).
Also, the average length of life of houses is extending. There are some different methods
for measuring ‘lifetime’, but most precise analysis was performed in Osaka, which showed the
ever increasing trend of it.
Average service
time (Years)
Statistic
Forecast
Chuoh Ward
Higashi-Yodogawa Ward
Hirakata City
, in Osaka Prefecture
Anno Domino
Figure-2 Trend of average lifetime of timber houses in 2 wards and 1 city in Osaka2)
That means, existing houses are expected to serve for generations of families through re-use,
or different families through resale of long-life house as stock. This situation is quite different
from that after 1945 until 1980s, where new construction and/or ‘scrap and build’ type
reconstruction of short-life houses were dominant, which served only for one generation of
‘core families’.
In order to cope with this new situation, the governmental housing policy pays more attention
not only to the quality (length of life) of houses, but also provision of reliable data/records of
houses attributed to physical buildings will assist the alteration and transactions of in future.
Record of houses and settlements are also valuable in case of disasters.
From the technical viewpoint, instead of conventional blue prints and written documents on
paper, digital data are becoming popular for synchronic communication. Usefulness of them
will further better in future. However, the coded data need media to record and replay, and
durability of current digital recording media is still insufficient to cover the whole lifetime of
houses. Also, data formats tend to change frequently, making the diachronic transfer of data
between ages suspicious.
2D images like drawings and photos could be at least printed on long-life papers and stored.
However, coded 3D data cannot be printed on papers as objects of description are seen. They
need drives and decoders to be replayed through computer graphics or three dimensional
printers etc. This nature of 3D data is common to such coded data as sounds and movies.
Therefore, we need durable recording devices, and method for assure the ‘readability’ of
various and changing formats of data for the length of time which is, at least, beyond the length
of life of physical buildings.
In order to strengthen technical basis for to cope with this new demand, ‘Basic Research on
Technologies for Permanent Archiving of 3D Data of Houses’ was performed by National
Institute for Land and Infrastructure Management (NILIM, in Tsukuba-science city) between
fiscal years 2012-2014.
This paper reports some alternative technologies we developed, toward the lasting use of 3D
data archiving houses and settlements.
1. Cases of organizations in charge of maintaining lasting data
Several representative organizations in Japan that are in charge of keeping important and
massive digital data were surveyed and technologies, methods and media which they use today
for storing ever increasing data were monitored. Many of them are keeping data in own data
centers, and carefully considering the out-sourcing of the activities, paying attention to the
security of the data.
(1) Hokkaido Building Guidance Center
The center provides charged service for filing and preserving data of individual houses since
2005, earliest among local authorities in Japan. The houses registered should meet the
standard for cold region, which is stricter than the minimum conditions required by the national
building standard law.
The data can contain digital files (CAD data, image data of drawings, photos etc.) up to
100MB. The data is saved in the data center (out-sourcing to the private firm), and also 3
pieces of CD-R are provided. These three disks are separately filed at the center, at the
contractor who built the house and at the owner of the house. The center selects a qualified
product of CD-R that is expected to last for more than ten years. When the ten years have
passed, the center sends mails to the contractors and house owners to refresh the media. They
say that this renewal procedure is also a good chance to make them recall the existence of such
archives.
The data center is located in the industrial complex near to Sapporo city, where average
temperature is rather low, which is good condition for data center. The data center is also
considering the usage of waste heat from the hard discs for regional air conditioning.
(2) National Diet Library
This organization keeps more than 38 million documents. Most of them are made of paper,
however, 670 thousands of recorded media and 8.9 million of documents in the form of micro
films are also filed.
Besides preservation of original documents, they are elaborating to create digital archives for
the purpose of disclosure, without physical damage to the original documents.
Sometimes books disclosed are attached with digital data in the form of diskette or CD-ROM.
For the purpose of reading, the library keeps machines and drives to read these legacy media.
(3) National Archives of Japan
Historical documents of the Japanese government are filed in this organization. Important
documents are scanned and disclosed in the form of digital archive, accessible through internet.
Annex storage is located in Tsukuba, c.a. 50 km from the main storage in Tokyo.
(4) Statistical Bureau, Ministry of Internal Affairs
National censuses are performed by this organization, and digital master data are coded and
filed in their own data center.
(5) Geography Survey of Japan
This institute is in charge of geographical survey, especially underground soil conditions and
mineral resources, and today under AIST, located in Tsukuba-city.
(6) Geospatial Information Authority of Japan
This institute is in charge of geographical survey and publishing various maps. Digital maps
(DM) are also available. In 2011, the damaged area of the great eastern Japan earthquake was
intensively surveyed and precise maps of changed land shape, damaged settlements were issued.
(7) Remote Sensing Technology Center (RESTEC)
Satellite image data are collected, kept and delivered from this institute.
(8) Building statistic
Building certification procedure is performed in Japan under control of Ministry of Land,
Infrastructure, Transport and Tourism(MLIT), and newly started construction works are
tabulated and issued from the Ministry of Land, Infrastructure, Transport and Tourism. The
master data is kept in a governmental data center and also in the form of MO (see 2(4)).
(9) Registered copyright of software/programs
Copyrights are registered in the Agency for Cultural Affairs, Especially computer programs
are registered in Software Information Center (SOFTIC). They had filed the registered works in
the form of micro films for lasting memory, however, started to receive digital data since June
2011. Up to march 2013, 11,557 programs have been registered.
(10) Public buildings
We monitored data on public buildings provided and kept by the local governments. In case
of Asahikawa city, CAD data of public offices, schools and rental houses are kept in a file server
maintained in the city hall.
(11) Technical office of MLIT (SXF)
Design drawings and completion drawing of national public works are recorded in a form of
standardized 2D data format named SXF. CAD data of infrastructure provided by consulting
firms are converted in the common SXF format and kept in the local technical offices of the
MLIT in the form of file server and CD-R, and the data is stored in the local technical offices
annex to the 8 provincial branch of the MLIT. They are summarized into GIS and utilized for
operation and maintenance works. This GIS data is managed by the NILIM.
(12) Japan Patent Office
Industrial Property Digital Library (IPDL) keeps digital data of patent and provides searching
services.
2. Historical review of media recording digital data
Vulnerability and risks of historical digital records are generally warned since 1990s.3)
From this viewpoint, we traced various memory devices which have been used to record
digital data (statistics, CAD, GIS etc.) in Building Research Institute (BRI since 1946) and
National Institute for Land and Infrastructure Institute (NILIM, established in 2001, partially
inherited those old recording media from BRI). They including MT, CMT, FD, MO, DAT,
HDD, CD-ROM, CF, SD etc. and we evaluated from viewpoint of durability.
In general, devices that keep the data in the form of electrons on silicon chips have the
shortest life, followed by magnetic devices. The devices that record the data in the form of
physical shape (e.g. punched papers and pressed CD-ROM, not CD-R) can last longest. The
durability of listed digital devices promised by the makers has not exceeded 100 years.
Beside the deterioration of recording media itself, the social life of the ‘drives’ will be also
problematic. Some of the media still correctly keep the recorded data; however, it has become
difficult to find the ‘drive’ to read out the data from the media, for example ‘5 inch MO disc’.
(1) Paper media for recording program/data
In 1970s, holes on paper ribbon or card was used to record program/data. Some of them still
remain in our institute for a kind of memory (Figure-3, 4). Through observation, the data
recorded on them are still readable through eyes or scanners. However, machines to read them
had disappeared (card reader, tape reader, etc.).
Figure-3 One piece of punched card
Figure-4 One series of punched cards containig statistical data
Also, some data printed on papers by ‘line printer’ in 1970s still remains (Figure-5). They
seem to be still readable through OCR.
Figure-5 Statistics output by line printers, blueprinted
(2) Magnetic media for recording program/data
Open-reel type magnetic tape was a media to keep statistic data in 1960-70s. Large tape
drive was equipped in main-frame computer rooms. Magnetic disc was also attached to
mini-computers in ‘80s. In those days, open-reel type magnetic tape was still used to record
sounds, however the specification was quite different and the tape decks for musical tape are not
useful to the tapes containing digital data.
In 1970s, compact cassette was introduced to record speech and music. Some of initial
micro-computers utilized this for recording and replaying digital data through ‘MODEM’
interface, which was also used to exchange data through telephone. Digital compact cassette
was used to record digital data for automation in factories. Tape players for compact cassettes
are still produced and sold in the market.
DAT, which was once popular for digital recording of audio, was also utilized as back-up
media for hard disc in 1990s. Today they are already obsolete.
Floppy discs that enable random access between tracks and sectors throughout a disc unit
were introduced in BRI (one of previous body of NILIM) in ‘80s. At first, the size was 8 inch,
and the density was 250KB. The recording capacity per disc was enhanced through usage of
both sides of a disc and doubling the recording density.
In 1990, satellite image data was delivered from RESTEC, in the form of 8inch floppy discs
in sequential recording format.
(5 inch double sided double density in MS-DOS format)
Figure-6 One of the most popular types of floppy disk
In order to regulate random access to multiple small files kept in a disc, DOS (disc operation
system) was introduced. At first, several DOS’s inherent to various systems (word processor,
games etc.) were used, and finally MS-DOS and Macintosh formats became main streams.
More compact 5 inches, and 3.5 inches types followed. The last 3.5 inch type had been used
until 2010. The 3.5 inch type MS-DOS discs were adopted as official digital media for
‘building certification’ procedure in 1990s.
In short, floppy disks had once flourished, and had disappeared rapidly, and during their era
(c.a. 20 years), their size, shape, density and format were very varied and rapidly changed,
making the future handling very difficult.
Table-2 Types of floppy disks, used in BRI 1980-2000
Diameter(inches) Sides
Density
Capacity
8
1
Single
250KB
8
2
Double
1.2MB
5
1
Single
160KB
5
1
Double
360KB
5
2
Double
640KB
5
2
Double
720KB
5
2
High
1.2MB
5
2
High
1.44MB
3.5
2
Double
640KB
3.5
2
Double
720KB
3.5
2
High
1.2MB
3.5
2
High
1.44MB
Sectors/Track
26
26
8
9
8
9
8
9
8
9
8
9
(4) Optical media for recording program/data
CD became popular in ‘80s for delivering recoded sounds, replacing the record discs (SP,
then LP). This was also used to deliver programs and data in mass scale, named CD-ROM.
In ‘90s, recordable CD-R was introduced to record and exchange large data up to c.a. 650MB.
This media is still widely used to keep the CAD data for public buildings and infra. The
expected length of life is assumed as 10 years. Production of CR-R media was out-reached to
developing countries, resulting cheap and uncertain (no-brand) products.
MO was also developed in Japan (5 inch, 3.5inch). 3.5 inch type is still used for recording
building statistics. MO was developed in Japan and did not become popular outside Japan;
however the quality is still stable. This media is mostly used in governmental offices and
private firms today in small amount. The recording density progressed from 160, 230, 640
MB etc., however, the latest drive can still read/write legacy media with lower density. The
round recording media is protected in plastic square housing, provided with metal shutter. The
recorded data is sensed without physical contact, similar to CD-ROM.
Figure-7 MO disk containing 3D data elaborated in 2000 (3.5 inch, 230MB format)
(5) Electronic media for recording program/data
In the beginning of IC memory chips, they were installed on the circuit boards of computers.
SRAM and DRAM were memories which forget all after power off. ROM (read only memory,
that can contain data without power supply) was also used for keep program for startup the
system. UVROM was re-writable, but need to be taken out from the circuit board for
programming while power is off. EEPROM can keep the memory during the power is off and
was the first re-writable memory on board.
Recordable ‘compact flash’ memory, which can keep the memory without power supply and
inserted into a standardized slot of machines, was introduced in ‘90s. It was widely used to
keep the image data obtained by digital camera. Interface slot/adapters for PC became popular,
in order to read the photos taken in fields. The device was also used to exchange the digital
data between PC’s. Soon, several alternative forms with similar chips inside were in sale (SD
etc.). Since 2000, similar memory devices provided with USB interface appeared in 2000’s. The
capacity of those devices has seemingly extended (20GB, etc), to keep digital photo images
with high resolution. However, the durability of the data is not secured. Especially, this type
is vulnerable against frequent re-writing access.
(6) Data center
Density of HDD (hard disk drives) rapidly developed from 20MB in 1990 to more than 1TB
(1,000,000MB) today. The loss of data caused by accidental crush was mitigated by RAID
technology through which broken drive can be replaced by new drive while data stored on them
is continuously kept and used. Such hard disks connected to local network is widely used to
store the data in offices today (file server, NAS etc.).
Since 2001, communal remote data center connected through internet provide service to keep
data. Physical device keeping data is hard disc drives, which rapidly enhanced the speed and
density. Data center is a kind of out-sourcing of maintenance works for file servers. Data
center connected to network is also useful to share the common data among remote users.
However, in case of archiving data of houses and buildings, the demand for access is not
frequent. The frequency of access might be only once in ten years. Consumption of
electricity is useless to keep such static data.
According to a survey performed by a Japanese think tank, total floor area of data center is
1.64 million square meters, and the total power consumption was 7.6 billion kWh in 2010.
They also estimated that it will reach 25% of total national power consumption in 20254).
3. Currently used data formats for modeling 3D objects
According to our survey, c.a. 300 data formats are used to record 3D objects. Documents on
the specification of major representative data formats are collected with sample data and images
of the sample objects, as listed in the table 2. Some of them (e.g. VRML) were authorized by
ISO; however, other data formats still followed and replaced/added. 3D data is useful to
describe the shapes of parts and structure of machines, widely produced by CAD systems and
used for CAM production lines.
Table-2 Major 3D data formats surveyed in fy.2010
Extension
3dm
3dmf
3ds
3dxml
ac
arc
ase
asm
atr
bdl
blend
blk
br4
c4d
cab
cadds
catdrawing, catshape
catpart, catproduct
cgr
chr
dae
ddf
dte
dem
df
dlv
drf
dwf
dwg
dws
dwt
Name
Rhino
Quickdraw 3D
3D Studio
3D XML
AC3D
I-DEAS
ASCII Scene Export
Pro/Engineer, Solid Edge, SolidWorks
Lightscape Material
OneSpace Designer
Blender
Lightscape Blocks
Bryce
Cinema 4D
TrueSpace
CADDS
CATIA V5
CATIA V5
CATIA Drawing
3Ds Max Characters
AutoDesk Collada
Data Descriptive File
DTED, National Geospatial-Intelligence Agency (NGA)'s Digital Terrain
Elevation Data
Digital Elevation Models
LightScape Parameter
CATIA V4
VIZ Reader
AutoDesk Composer Design Web Format
Legacy AutoCAD Drawing
AutoCAD Standards
AutoCAD Drawing Template
dxf
e00
eim
exp
fac
fbx
fbl
fig
flt
fmz
ft#, rt#
gmax
geo, scn
gml
gts
hrc
htr
ifc
ige, igs, iges
ini
iob
ipt, iam
iv
jt
k3d
kmz
lay
lp
ls
lw
lwo
lws
lxo
m3g
ma
max
mb
map
md2
md3
mdd
mel
mf1
model
mot
mp
ms3d
mtx
ndo
neu
Ntf
obj
obp
off
par, psm, pwd
pdb
pd, pd
pkg
plt
ply
pov
pp2
prc, prd
prw
prj
prt
pwc
pz3
raw
rib
rif
rvt, rte, rfa
rwx
s3d
sab, sat
scn
sda, sdp, sds, sdw, ses
sdac, sdpc, sdsc, sdwc
session
sfc
shp, shx, dfb, prj
skp
slc, sl, slo
sldasm, sldlfp, sldprt
slp
stl
stp, step
svg
tab, dat, id, map
u3d
unv
vrml
vue
vw
w3d
wings
wire
wrl, wrz
x3d, x3dv
AutoCAD Drawing Exchange Format
TIGER/Line Arc Interchange format
Electric Image
CATIA V4
Electric Image
AutoDesk Kaydara FBX
CADfix Log File
xfig
Flight Studio OpenFlight
FormZ Project File
TIGER, Topologically Integrated Geographic Encoding and Referencing
AutoDesk Game Creator
LSS
OpenGIS Geography Markup Language
GNU Triangulated Surface
SoftImage
Motion Analysis HTR file
Industry Foundation Classes
Initial 2D/3D Graphics Exchange Specification
POV-Ray animation script
3D Object TDDDB Format
AutoDesk Inventor
Open Inventor
JT
K-3D Native
Google Earth Model
LightScape Layers
LightScape Presentation
LightScape
LightWave 3D
LightWave 3D 5.0 Object
LightWave 3D Scene
Luxology Modo
JSR-184
Maya Scene ASCII
3Ds Max
Maya Scene binary
Quake 3
Quake 2 Player Model
Quake 3
Vertex Key Frame Animation
Maya Embedded Language Script
I-DEAS
CATIA V4
LightWave 3D Motion
Maya Scene PLE
MilkShape 3D
OpenFX Model
Nendo
Pro/Engineer
National Transfer Format
Wavefront
Bryce
DEC Object file
Solid Edge
PDB Reader v3
CADDS
I-DEAS, OneSpace Designer
HP-GL
Stanford PLY
POV-Ray Rendering Instructions
Poser
PRC Adobe 3D Reviewer
Adobe 3D Reviewer
3Ds Studio Mesh
I-DEAS, NX (Unigraphics), Pro/Engineer
Pulse
Poser
Raw Faces
Renderman
Radiance
Rent
Renderware
Strata 3D
ACIS
TrueSpace
OneSpace Designer
CATIA V4
OpenGIS Simple Features for CORBA format
Shapefile, Esri's open hybrid vector data format
Google SketchUp
Renderman
SolidWorks
Pro Engineering
Stereo Lithography
Standard for the Exchange for Product Data
Scalable Vector Graphics
MapInfo's vector data format
Universal 3D
I-DEAS
Virtual Reality Modeling Language
AutoDesk 3D Studio Animation
LightScape View
Shockwave 3D Scene
Wings 3D
Alias Wire
VRML 1.0, VRML 2.0, VRML 97
Extensible 3D (VRML 3.0, uses XML, 2004)
x
x b, x t, xmt, xmt txt
xas, xpr
Xyz
Direct X
Parasolid
Pro/Engineer
Simple point cloud formart
BRI also started to utilize 3D data of houses and settlements in 1993 for the purpose of
landscape designing. Initial core part of the system ‘Landscape Simulator’ was developed
during 1993-96 and started to be delivered through internet as ‘free ware’ since 1996. For this
system, in 1995, a native data format (LSS-G(1)) was defined and programs were coded in C
language to input and output in the format, which was packaged in ‘IP’ library (a kind of
interpreter for LSS-G commands).
The latest version 2.09 was released in March 2011, and modeling functions for creating flat
site for housing complexes on the hilly land are added in 2012. However, the data format has
not been changed since the original version for 17 years.
In the first stage before 1996, modeling functions of the system were very limited, therefore,
3D data of houses and settlements were provided by professional draft man operating
commercial CAD system, and then imported to the scene through file converters. Several file
converters were elaborated to import external 3D data (Dxf, Microstation, Minicad, Vector
Works, VRML, SHP etc.), coded in C language.
Modeling of existing houses from photos was also often performed (.skv format). Land
shape was also imported through file converter from DEM style data provided by Geography
Survey Institute in 2 formats (.mem:50m mesh, .lem:5m mesh).
Since beginning of 21th century, mushrooming of new formats occurred. Many of them are
XML based. Usually, attributes are defined on 3D objects in various styles.
Exporting/importing data of houses and settlements between different systems became
inevitable for simulating various issues. Examples are urban scale fire simulation and
evacuation in case of emergency, etc. Evaluation of settlements from viewpoint of global
environmental issues was also required.
In developing countries, CAD software without
strict license control was affordable for students, and widely used to produce 3D data.
In 2004, in the field of civil engineering, standard data format (SXF) for digital submission of
designed data (CAD) was established by NILIM. Result of designing social infrastructure
(bridges, tunnels, pavement, banks etc.) are converted into this format, and handed to the branch
offices of the ministry.
The SXF format specifies only 2D vector data, and 3D format is still under active discussion.
However 2D digital data is far easier than the conventional drawings printed on papers for
modeling designed 3D objects.
Very recently 3D printers became cheap and popular. They require specific data formats.
Also, displaying 3D objects including houses as augmented reality (AR) or mixed reality (MR)
using popular tablet devices is useful.
Usually, adaptation to this situation is performed by applications through providing file I/O
functions in the form of ‘plug-in’, or through provision of external files converters among major
contemporary file formats.
However, from viewpoint of diachronic longevity of data, when a format will become
obsolete in future, re-use the legacy data will be difficult. File conversion to contemporary file
formats should have been done before the data format is forgotten. If once a file format is
forgotten, the data describing an important building cannot be opened. Even an application
executable is preserved; the environment to execute the program might be already lost.
For example, if we have to re-use a file created by legacy CAD system and saved in an 8 inch
floppy disc, double sided high density (2HD), formatted by MS-DOS, then we need following
items and steps:
(1) Disc drive which can read the media
(2) PC to which the drive can be connected
(3) OS accompanied by device driver to operate the disc drive (Windows 95 etc.)
(4) Historical CAD system that can open the data file
In order to perform all of these, we need a kind of zoo rather than museum to retain antique
machines and software at live status.
Then we can open the file which is recorded in the diskette (if it is not broken), and observe
the shape of building described in the file. However difficulties still remains for us to take out
the data in some contemporary file format, and re-use it.
If the hardware (PC and disc drive) and software (DOS and CAD system) are not available,
then following alternative ways should be taken.
(1) Read all the physical tracks and sectors of the floppy disc, neglecting the read-error that may
occur, and save it in a long one file in binary format.
(2) Analyze the organization of file system (DOS), if file allocation table is not broken.
(3) Take out the file of data which describe the desired houses.
(4) Provide file converter, based on some historical document on the CAD system at that time,
or directly analyze the file format
Some of these problems will be solved, if we could describe (2)-(4) in a virtual environment,
in which functions of disc drive, dos/OS, cad, display etc could be described in scripts. Virtual
zoo of antique machines on virtual machines will be cheaper to maintain than physical zoo of
machines and programs.
However, if all the information on the format of the data is missing, we need analysis with try
and errors. That often occurs also, if the data is given in a format which is popular but not
disclosed in an available document (kept secret). In such case, we have to look into the inside
of the data.
Fortunately, in case of 3D data, the result of the analysis will be proved through the
successful 3D presentation on the display by using some contemporary software. The
achievement of such archaeological research will be recorded in the form of ‘source code’ of
file converters.
It suggests that alternative technologies are needed, which enables keeping data for a long
term with intension. The expected term will be longer than the lifetime of houses, with less
consumption of energy (electricity), and less risk of unexpected deletion/loss/alteration,. The
demand for accessing speed is not the issue, in comparison to the contemporary ‘big data’ on
‘crowd storage’.
Also, the risk of data leak or illegal alteration remains if the data are kept in hard disk drives
which are always connected to network.
4. Development of alternative technologies
(1) Static and sustainable media for recording
In the history of architecture and buildings, record of construction attached to a building
played important role to identify the absolute year of construction. Usually, name of
carpenters are also recorded on the pentagonal timber plate.
Figure-8 Historical ‘MUNAFUDA’
Figure-9 Electronic recording board, we propose
Even though the amount of data saved on such device is very limited (<1kb), the length of the
life of documents written on the timber plate with traditional black ink is quite long. This
traditional recording media is called ‘Munafuda’ (a timber plate attached under highest beam of
a building recording the date of construction and name of client and carpenters), can last longer
than documents on papers. The oldest one dates in 1122 kept in Chuson-ji temple (world
heritage) is estimated to have been attached to a lost library building for storing literature of
Buddhism5). Documents written on timber plate are called ‘Mokkan’ in general, which are
often excavated from ruins after 7th century.
Conditions required for archiving 3D data describing houses and settlements are significantly
different from that for synchronic transaction/exchange of 3D data for business.
- Accessing speed is not essential (several days are allowed to access to the old data)
- Frequency of access is very low (maybe once in ten years)
- Cost/energy for keeping is important (hopefully free from power supply)
- Security against illegal alteration/deletion is important
In order to meet these requirements, we chose and proposed ‘curving on long-life material’.
Recently fine laser cutters are available, enabling fine (middle dense) recording on lasting
materials, like stones. The curving can be performed in small firms. The curved data will be
scanned, and the current/future OCR technologies can easily pick up the scanned data.
The curved data contains (a) 3D data itself, (b) a metafile to describe the format (for virtual
converter), and (c) some additional small images and short literal explanation.
(2) Metafile handling to assure readability
Since 1980s, efforts to standardize digital data describing buildings and built-up environment
have been made. Exchange of data between different CAD systems was desired. Fortunately
these problems have been seemingly improved.
Useful software that can create 3D models appeared, like SHADE, ArchiCAD, Myhome
Designer, Studio 3D Max etc. have been developed. Delivery of 3D data through internet was
also enabled through introduction of common DXF, VRML etc.
All those efforts were directed to the contemporary data exchange between different systems.
If a final ultimate permanent common data format and recording media could be defined and
agreed globally, then it would be enough for us to convert any data into it, and store it forever.
If it is difficult, then the second choice is to continue efforts to find more useful technologies,
and ever converting the data into the contemporary most popular container at that time.
This research tried to find another, the third approach.
3D data is saved in the form of a file, which is equivalent to a bit stream (0110110000), and
the preservation of any 3D data is subdivided into 2 issues, namely:
a. Media to keep the bit stream
b. Format of the bit stream
A metafile attached to data is useful to describe how the data describe a house or a settlement.
We developed a compiler system, named ‘virtual converter’, which compiles the metafile and
create an executable which will input the data file, parsing the grammar, and get elements that
describe the house or the settlement.
<Recording phase today>
[1] When we preserve a file that record a house, we attach a metafile.
[2] We record the set of data file and attached metafile on durable material
<Replaying phase in future> (maybe 100 years later)
[3] The bit stream containing data file and metafile is obtained from the recording material
[4] The metafile is compiled by virtual converter, creating an executable
[5] The executable will open the data file and do actions for usage
The metafile in this system looks like a kind of program for replaying the data file. The
metafile need not cover all the specifications of a format (e.g. whole features of dxf) suggested
by the extension of data (.dxf) and omit the features that are not referred by the data. However,
if some additional dialect or misinterpretation of the specification is performed in the data, it
must be followed in the description of metafile, if the feature is significant to preserve.
The grammar of the metafile we developed and proposed is a kind of programming
language(2) to be processed by a compiler system. The grammar has following features:
[1] A metafile is described by ASCII characters.
[2] The grammar of metafile is similar to ANSI-C language, but simpler
[3] The metafile can refer to library functions to define 3D data
[4] The library functions contains LSS-G commands as subset
[5] The library also provides commands for scanning file, and accessing to 3D data in memory
Based on this grammar, we developed a new system of ‘virtual converter’, a kind of compiler
system, which accepts and processes a pair of 3D data and attached metafile. This compiler will
read the metafile and creates a run-time program, which can decode the 3D data in future.
The sample applications we provided with the ‘virtual converter’ are (a) file-file converter,
(b) virtual reality display, (c) tablet AR viewer (android based) and (d) web based database entry
(server based), described in the (3). These are not needed directly when the data is saved.
However they are useful to check and debug the metafile to make sure the future readability.
The definition of the grammar (version ID) for creating the metafile must be also attached to
the preserved data of each house. However, it is enough to keep the grammar book in e.g.
national library, and data for each house refers to the ID of the book.
Even if some of the rules for creating metafiles are changed in future, the revised version of
definitions will be stored with different ID, and the change will not affect the ‘readability’ of
previous data file and metafile.
Theoretically, the developed method is inspired by the concept of compiler-compiler
proposed by Futamura (1971) (3).
(3) Prospect of usage in future
Of course, we cannot clearly predict the needs and usage of preserved 3D data in future;
however, it is worth while assuming some typical usages which are already practical today.
For this reason, we developed 4 trial utilizing systems that can input pair of data file and
metafile and execute something. All these programs/systems contain the same compilers that
handle the pair of data file and metafile, while different linked libraries perform different way of
utilization, providing different actions triggered by the same library commands (COORD etc.).
(3-1) File-file converter
The compiler is triggered by the command line console with three string parameters, which
compiles the metafile (first parameter), and the executable (running on the virtual machine)
input the data file (second parameter), and finally output a file with the name given as the third
parameter. The current version (vc-1c.exe) is developed on VS2005 as console application,
and executed on cmd.exe on Windows versions. This version processes any file format
defined in the metafile, and library functions write a file in LSS-G format. This simple
application was used to develop, test and debug the prototype kernel of the compiler.
Figure-10 Command line (to convert a metafile In_vrml.cmm and a data file house1.wrl into house1.geo)
(3-2) Presentation by virtual reality devices
This system receives a pair of metafile and data file, and the obtained 3D building is
presented as virtual reality (stereoscopic view). The system compiles the metafile, and the
resulted executable input the data file and synthesize the result as 3D objects in the memory of
the system, which are presented on the stereoscopic display.
1. Title: Data entry to virtual reality presenter
2. Selection of metafile
3. Selection of data file
4. Start operation
Figure-11 Converting a pair of metafile skv_i.cmm and data file Z(C6_4).geo
Figure-12 Presenting by the VR projector
(3-3) AR viewing in the field
The way of usage of preserved 3D data is almost same as (3-2). However this system runs on
tablet devices and obtained 3D objects was synthesized on the photo image obtained by an
image censor equipped on the back of the device, and the synthesized image is displayed on the
LCD of the device. The tested device was ‘Android’ based, instead of ‘Windows’ based (3-2).
Therefore, for this system, the source code of ‘virtual converter’, which was developed on
VS2005 for Windows, was re-compiled on Android OS by using ‘ndk’ compiler system. The
compiled ‘virtual converter’ linked with wrapper application written in Java language run
immediately on Android based tablet and almost compatible to that on Windows system(4).
Figure-13, 14 AR view of a pair of metafile and data file by tablet
(3-4) Data storage and conversion on web-server with SQL database
This trial system runs on the web-server, with large data storage in SQL server, which
contains two different dialogs accessible through network.
1. Title: 3D data storage
2. History of past operation (upload/download)
3. Entry to upload page
4. Entry to download page
Figure-15 Entry point of 3D data storage – web site
A. Upload function
Users can assign two local files, namely data file and metafile, and request ‘upload’ function.
The server receives two files, and compiles the metafile. The created executable processes the
data file and the obtained elements (coordinates, vertexes, lines, faces, solids, etc.) will be stored
in the SQL server in the form of tables. This process need far more time than system
(3-1)-(3-3). Therefore, the process is performed in the backyard, as a process separate from
data accepting procedure. The progress of the process is shown to the client through
individual web page created for each upload event. The contents visible at client is
automatically updated when the status.
1. Title: 3D data storage
2. Subtitle: Upload
3. Selection of local metafile
4. Selection of local data file
5. Start uploading
Figure-16 Uploading a pair of metafile and data file through web page
B. Download function
Users can select uploaded 3D data from the list, and assign a metafile that defines the data
format to be downloaded. The executable will select and pick up elements from SQL tables,
and synthesize the new 3D file defined by the metafile. This process also take more time
equivalent to process A. Therefore, the progress is reported to the users until download is
completed.
The users can quit the contact to the web server and shutdown the client terminal during the
process. Accessing to the page again lets download the result after the completion.
Process A and B are taken at the same time, when a user desire to utilize some preserved old
data file and metafile through conversion to contemporary format. However, if desired, the
SQL tables created by the upload process can be maintained before the future download process,
and SQL tables (physically files on HDD) can be utilized for data storage for a short time (<10
years).
1. Title: 3D data storage
2. Subtitle: Download
3. Selection of 3D data in storage
4. Selection of local metafile
5. Start dowloading
Figure-17 Downloading a data file in any format defined by respective metafile
(4) Possibilities of other alternatives
We realized a system of handling data file and metafile, in order to keep the readability of
data for a long time. We neither protest that the system we proposed is the only one system,
nor the best system to achieve the purpose. Various other systems for the same purpose can be
developed.
In order to preserve a data, we attach a metafile (1) based on the compiler (1). The
definition of the compiler is printed on a book and kept in a library in charge of lasting data
preservation, and the future user can compile the metafile and then use the data.
Assume that another compiler (2) is developed and another metafile (2) will fulfill the same
purpose. We can attach both metafile (1) and metafile (2) to the original data, the future user
can use both compiler (1) and compiler (2) as alternatives.
If the compiler (2) performs better than the compiler (1), then those who wish to preserve
data will choose compiler (2) to provide metafile, and the users of the compile (1) will decrease.
Even in such case, the readability of previous data attached with metafile (1) will be assured in
future. It means that the two compilers (1) and (2) are not competitive, but co-operative.
We expect that a metafile is provided and attached to the data file when it is saved and
preserved in the form of durable media. However, later generations can also analyze a data file
described in an unknown format and describe the result in the form of a metafile. Since then,
the data file will become readable as well as a data file which was attached metafile since
beginning. It means that this compiler-metafile system can play a role of tools for
archeological studies for digital data.
The most important thing is that at least one compiler exists and is available, to provide a
metafile to be attached to the data file.
5. Next steps
Technical approaches toward practical production of the developed electronic recording board
at appropriate cost will be the main issue. During this experimental stage, sample data will be
obtained from past houses that are already lost but requested to be recorded forever and ready to
be disclosed.
(1) Indexing past/conventional 2D data into 3D archives
Modeling 3D data from old 2D analogue records (mostly papers) and utilize 3D model for
searching and indexing original historical records are being challenged. Data of individual
houses are collected into settlement model, and the source data for modeling (photos, maps and
drawings etc.) will be connected to the 3D model of the lost settlements.
(2) Preparing for expected usage in future
Simulations will be performed on settlements recalled from archives in desired formats. For
that purposes, various metafiles will be elaborated based on the grammar, so that downloading
from the server with any desired format for simulating will be obtained.
Conclusion
Today, hard disk drive (HDD) is the major device to keep the massive digital data. In order
to secure the risk of crush, RAID technologies are utilized, and the out sourcing of data center
management was started in the beginning of 21c (called ASP, then ‘cloud’). However, huge
consumption of electricity (30Giga Watts to operate the world’s data centers, equivalent to 30
atomic plants) for drives to keep the ever increasing digital data seems to be unsustainable.
In order to conserve the data which record past houses and settlements, for future generations
with small energy consumption, infrequent access and high security, the method which we
developed and proposed in this paper will provide alternatives.
Notes:
(1) LSS-G format (1995) is defined in the NILIM’s web site6), and also explained in a printed report7).
Library functions available for the Metafile include LSS-G commands as a subset.
(2) Summary of the grammar for the metafile is as follows:
・Numerical types: 32 bit integer, 32 bit float, 256 bit quaternion
Used for definition of valuables, functions.
・Functions
Called from other functions
The main function can be omitted, when no sub-functions are used.
・Program flow control
if(condition){・・・}[else{・・・}]
do{・・・}while(condition)
while(condition){・・・}
for(initialize; condition; incremental{・・・}
switch (condition){case }
・Library (built-in) functions
data scanning from data file
data acquisition from SQL server or Memory Data (same command)
definition of coordinate, color, vertex
definition of line, face
definition of significant group of lines or faces (solid body, group of parts)
(3) Futamura8) argued concept of compiler-compiler from viewpoint of partial evaluation in 1971.
program
A
π(C,R)
processes constant value C and runtime value R, compiler α converts the program π into
α(π,C) (R)
which process only the runtime value R, and the process becomes simpler.
For example, if π is ‘Y=1+C+R’ where C = 2, then α(π,C) is ‘Y=3+R’.
If πis an interpreter int, that executes source code S with runtime data R, the program int(S,R) is
partially evaluated by the compiler to become a programα(int, S)(R) that process only R. (projection 1)
Similarly, the relation α(int, S)(R) = α(α, int)(S)(R) is derived, where α(α,int) is a compiler that
translates the source code S to make a runtime program which process the data R. (projection 2)
Further, α(α, α)(int) is derived which translates the interpreter ‘int’ into compiler where α
functions as ‘compiler-compiler’ (projection 3)
FILE command in the LSS-G(3) format (1995) for defining models for landscape simulation performs
similar. For example, a command
G=FILE(B.geo);
in an LSS-G file A.geo refers to another file ‘B.geo’, and put a fixed model B as a part of the model A.
If the FILE command refers to an executable E, with runtime data R
G=FILE(E.exe, R);
then the executable E.exe is called with runtime data R and the result of the calculation is put as a part of
model A. For example, E.exe will create a door which is half open (parametric definition of a model),
and R gives the rate of open, then the fixed shape of a door open 30% will be resulted. In the same way,
program C is a converter to import a model F.dxf described in a foreign data format, then
G=FILE(C.exe, F.dxf);
means a model imported (converted) from F.dxf. (projection 1)
In those cases, E.exe or C.exe are native to MS-DOS or Windows environment, and cannot be used on
other OS like UNIX or Macintosh.
However if VC is a compiler which compiles the source code C.src and create a runtime program
whose performance is similar to C.exe,
G=FILE(VC, C.src, F.dxf); (projection 2)
performs equivalent to
G=FILE(C.exe, F.dxf);
on MS-DOS system, and if VC for another operating system is provided, C.src is commonly used on it.
If a common compiler-compiler CC is provided for various operating systems, and the source code of
the VC.src is provided, the executable VC is commonly used among those operating systems, and the
command will be equivalent to:
G=FILE(CC, VC.src, C.src, F.dxf);
If we use a multi-platform compiler system CC which is compatible among various operation systems,
and define VC.src for that compiler as a permanent agreement as a grammar for metafile C.src, the
definition of a converter C.src will be useful forever in order to process F.dxf. G=FILE(CC, GRAMMAR,
METAFILE, DATAFILE); (projection 3)
Once the compiler-compilers CC native to various platforms process that can generate an executable of
virtual converter defined by GRAMMAR, then the executable process METAFILE to create the
converter’s executable that input the DATAFILE.
However, the purpose of this system is the readability and longevity of digital data, instead of the
execution speed which was the basic concept of Futamura.
(4) Minor difference of compilers between VS2005 for Windows and NDK for Android was identified in
the handling of address calculation of arrays, minor difference between Windows (VS2005) and Android
(ndk) was identified. For example,
Integer array and variables
int stack[1000];
int stp;
are defined as global variables, and following command is executed.
stack [stp] = f1(stack[stp]);
If the value of stp changes in the function f1(),
executable created by VS2005 save the return value into the address, using the new stp value after the
execution of f1 function, while Android version save it to the address pointed by the stp value before the
execution of f1 function.
This kind of problems (compatibility) can be solved by extracting the program into two lines:
int stack[1000];
int stp, return value;
return value = f1(stack[stp]);
stack[stp] = return value;
Reference:
1) Statistics Bureau. Ministry of Internal Affairs and Communication. “Quick Summary Report of Census
on Housing and Land 2008” (in Japanese). http://www.stat.go.jp/data/jyutaku/2008/10_1.htm (accessed
2013-7-1)
2) Tsutsumi et-al. A Study of Trend for Stock and Demolition of Wooden Dwelling House. Journal of
Architecture and Planning No.649, Architectural Institute of Japan, 2010.5, 695～700 (in Japanese)
3) Rothernberg J. ‘Ensuring the Longevity of Digital Information’ 1999, RAND,
http://www.clir.org/pubs/archives/ensuring.pdf (accessed 2013-7-1)
4) MIC Research Institute. ‘Survey on power consumption by data centers toward green IT’ 2010, in
Japanese
5) IWATE NIPPO CO., LTD. (A local press company in Iwate prefecture) Memory of past glory of
Hiraizumi: the world heritage (in Japanese). http://www.iwate-np.co.jp/sekai/sekaiisan/sekaiisan4.htm
(accessed 2013-7-1).
6) Ministry of Construction (1998) Description of Data Structure of Landscape Simulation System (in
Japanese). http://sim.nilim.go.jp/VRNCS/MANUALS/Man-STR/INDEX.HTM (accessed 2013-7-1)
7) Kobayashi, H. ‘The Architecture of Landscape Simulation System Ver.2.09, provided by MLIT’
Research Report of National Institute for Land and Infrastructure Management No.42, (in Japanese),
2011.3, p.53-64, http://www.nilim.go.jp/lab/bcg/siryou/rpn/rpn0042.htm (accessed 2013-7-1)
8) Futamura, Y. “Partial Evaluation of Computation Process-an Approach to a Compiler-Compiler-“,
Systems, Computers, Controls" Vol.2, No.5, 1971, p45-50