Detecting figures and part labels in patents: competition-based development of graphics recognition algorithms (original) (raw)
Related papers
2014
We report the findings of a month-long online competition in which participants developed algorithms for augmenting the digital version of patent documents published by the United States Patent and Trademark Office (USPTO). The goal was to detect figures and part labels in U.S. patent drawing pages. The challenge drew 232 teams of two, of which 70 teams (30%) submitted solutions. Collectively, teams submitted 1,797 solutions that were compiled on the competition servers. Participants reported spending an average of 63 hours developing their solutions, resulting in a total of 5,591 hours of development time. A manually labeled dataset of 306 patents was used for training, online system tests, and evaluation. The design and performance of the top-5 systems are presented, along with a system developed after the competition which illustrates that winning teams produced near state-of-the-art results under strict time and computation constraints. For the 1st place system, the harmonic mean of recall and precision (f-measure) was 88.57% for figure region detection, 78.81% for figure regions with correctly recognized figure titles, and 70.98% for part label detection and character recognition. Data and software from the competition are available through the online UCI Machine Learning repository to inspire follow-on work by the image processing community.
2016
Abstract—We report the findings of a month-long online competition in which participants developed algorithms for augmenting the digital version of patent documents published by the United States Patent and Trademark Office (USPTO). The goal was to detect figures and part labels in U.S. patent drawing pages. The challenge drew 232 teams of two, of which 70 teams (30%) submitted solutions. Collectively, teams submitted 1,797 solutions that were compiled on the competition servers. Participants reported spending an average of 63 hours developing their solutions, resulting in a total of 5,591 hours of development time. A manually labeled dataset of 306 patents was used for training, online system tests, and evaluation. The design and performance of the top-5 systems are presented, along with a system developed after the competition which illustrates that winning teams produced near state-of-the-art results under strict time and computation constraints. For the 1st place system, the har...
Figure detection and part label extraction from patent drawing images
The US Patent and Trademark Office, together with the NASA Tournament Lab, launched a contest to develop specialized algorithms to help bring the seven million patents presently in the patent archive into the digital age. The contest was hosted by TopCoder.com, the largest competitive online software developer community. The challenge was to detect, segment and recognize figures, captions and part labels from patent drawing images. The solution presented in this work was the winning submission.
The aim of this document is to describe the methods we used in the Patent Image Classification and Image-based Patent Retrieval tasks of the Clef-IP 2011 track. The patent image classification task consisted in categorizing patent images into pre-defined categories such as abstract drawing, graph, flowchart, table, etc. Our main aim in participating in this sub-task was to test how our image categorizer performs on this type of categorization problem. Therefore, we used SIFT-like local orientation histograms as low level features and on the top of that we built a visual vocabularies specific to patent images using Gaussian mixture model (GMM). This allowed us to represent images with Fisher Vectors and to use linear classifiers to train one-versus-all classifiers. As the results show, we obtain very good classification performance. Concerning the Image-based Patent Retrieval task, we kept the same image repre-sentation as for the Image Classification task and used dot product as sim...
2013
To verify the originality of an invention in a patent, the graphical description available in the form of patent drawings often plays a critical role. This paper introduces the importance, requirements and challenges of a patent image retrieval system. We present a brief account of the work done in the specific and related areas of the patent image domain. We begin with a review of work done dealing specifically with retrieval and analysis of images in the patent domain. Although the literature found dealing with patent images is small, there is a significant amount of work that has been done in related areas that is useful and applicable to the patent image area. From a methodological point of view, we present an overview of the algorithms developed for the retrieval and analysis of CAD and technical drawings, diagrams, data flow diagrams, circuit diagrams, data charts, flow charts, plots and symbol recognition.
Caption-guided patent image segmentation
Proceedings of the 2016 Federated Conference on Computer Science and Information Systems, 2016
The paper presents a method of splitting patent drawings into subimages. For the image based patent retrieval and automatic document understanding it is required to use the individual subimages that are referenced in the text of a patent document. Our method utilizes the fact that subimages have their individual captions inscribed into the compound image. To find the approximate positions of subimages, first the specific captions are localized. Then subimages are found using the empirical rules concerning the relative positions of connected components to the subimage captions. These rules are based on the common sense observation that distances between connected components belonging to the same subimage are smaller than distances between connected components belonging to various subimages and that captions are located close to the corresponding subimages. Alternatively, the image segmentation can be defined as a specific optimization problem, that is aimed on maximizing the gaps between hypothetical subimages while preserving their relations to corresponding captions. The proposed segmentation method can be treated as the approximate solution of this problem.
Image search in patents: a review
International Journal on Document Analysis and Recognition (IJDAR), 2012
To verify the originality of an invention in a patent, the graphical description available in the form of patent drawings often plays a critical role. This paper introduces the importance, requirements and challenges of a patent image retrieval system. We present a brief account of the work done in the specific and related areas of the patent image domain. We begin with a review of work done dealing specifically with retrieval and analysis of images in the patent domain. Although the literature found dealing with patent images is small, there is a significant amount of work that has been done in related areas that is useful and applicable to the patent image area. From a methodological point of view, we present an overview of the algorithms developed for the retrieval and analysis of CAD and technical drawings, diagrams, data flow diagrams, circuit diagrams, data charts, flow charts, plots and symbol recognition.
Towards content-based patent image retrieval: A framework perspective
World Patent Information, 2010
In this article, we discuss the potential benefits, the requirements and the challenges involved in patent image retrieval and subsequently, we propose a framework that encompasses advanced image analysis and indexing techniques to address the need for content-based patent image search and retrieval. The proposed framework involves the application of document image pre-processing, image feature and textual metadata extraction in order to support effectively content-based image retrieval in the patent domain. To evaluate the capabilities of our proposal, we implemented a patent image search engine. Results based on a series of interaction modes, comparison with existing systems and a quantitative evaluation of our engine provide evidence that image processing and indexing technologies are currently sufficiently mature to be integrated in real-world patent retrieval applications.
Visual Structure Analysis of Flow Charts in Patent Images
This report presents the work carried out for the flow chart recognition task in the course of the CLEF-IP 2012 competition. The goal is to obtain structural information of flow charts based on the visual content of the images. To this end, for each flow chart a list of its nodes and their interconnections, i.e. its edges, is extracted and the type of the nodes and edges and attached text is recognized. The automatic recognition task is done in three stages: (1) flow chart image pre-processing using connected component analysis, morphological filters and line segmentation, (2) identification of nodes, junction points, end points and edges and (3) recognition of text, geometric node types and edge directions. Examples demonstrate good recognition results obtained for 100 tested flow chart images.