FEEDBACK RESEARCH PROCESS: FINDING YOUR RESEARCH FASTER AND MORE EFFECTIVELY

Introduction

When conducting a systematic literature review, one may begin the research using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). An example of the basic PRISMA research process is shown in Exhibit 1 – PRISMA Research Flow. The PRISMA process is used in many systematic literature reviews and is required in many healthcare journals (Liberati et al., 2009). While not required in most research fields, this approach, research process, and methodology provide a mechanism for in-depth research. The PRISMA approach requires extensive documentation of search results and rigid documentation but does not require analytical processing to complete. It is a painstaking and tedious process that may not be wholly appropriate for the broad subject of all research. However, it provides a significant starting point and base set of broad referenced work.

If someone starts research from scratch, they are likely new to doing research or conducting research in a new field of study. If one is an expert in their field, they likely have a database of information at their disposal. Experienced researchers gain an affinity toward journals they value, authors they have cited in the past, and people they have developed relationships with during their time conducting research.

Starting from nothing with literature research can be daunting, tedious, laborious, and painstaking to find the desired articles that meet the specifications and research topics. Standard research takes a path of searching for articles, reading abstracts, filtering, reading articles, and determining the article’s relevance. The issue today is too much information. For example, navigating Google Scholar and searching for agile software development returns 467,000 results. Searching for globally distributed software development returns 2,390,000 results. How does one find the proverbial needle in a haystack with this many search results? How many pages deep in the search results does one traverse before declaring the search for titles and abstracts of interest at an end? There must be a faster and more effective manner to find the desired research.

Research Processes

Backward Citation Expansion

Many start their research by fighting through the vast sea of information in Web of Science, EBSCOhost, JSTOR, or Google Scholar searches. They scan titles from the search results, select potential candidates, and skim the abstract. Next, they read the articles if the abstract seems viable. When one finds an article of interest, they jump to the reference section, beginning a similar process over again. They scan the reference titles for potential articles of interest. When interesting articles are found, the analytical process begins again with reading the abstract. The method of utilizing the citations in an article and the reference section it identifies articles of interest is known as Backward Citation Expansion (Briscoe, Bethel, & Rogers, 2020). Backward Citation Expansion is depicted in Exhibit 2 below.

Backward citation expansion provides a clear mechanism to broaden one’s research. Expanding out typically finds older articles because one started with the newest work. This is a look back in time. Exhibit 3 – Node Connection Diagram shows how backward citation expansion broadens the research. Starting from the central node, each reference that is followed creates a new node. The new node, in turn, has references that lead to additional articles.

Forward Citation Expansion

Forward Citation Expansion drives the research to help find the newest article on the topic. Forward Citation Expansion is considerably harder to complete. One would look up all the citations for an article and follow the path forward to an article of interest (Chen, 2018). Forward Citation Expansion would typically be accomplished by searching a bibliographic database (Briscoe et al., 2020; Chen, Lin, & Zhu, 2006). Exhibit 3 – Node Connection Diagram shows how forward citation expansion focuses the research by linking the older research to the newer referred citation. Each linked citation leads to a new article from an outer node toward the inner-most node.

Feedback Research Process

Using either Forward Citation Expansion or Backward Citation expansion can significantly broaden the research on a specific topic. The research methods described above are slow, require extensive reading and re-reading of each article, and do not yield overall results quickly. A more practical approach needs to be considered with Google Scholar results being returned in the millions of items.

Beginning with Backward Citation Expansion with the desire to determine a complete answer to the research question iteratively led to this new literature research process. Each article was scanned for topical research applicability. When accepted, the article is placed in a reference database for logging, future referencing, and tracking. For each article, quotes and notes were also collected for usage in future work. While this pulls data out of the article, it does not move the research forward from a systematic literature perspective. In order to move the research forward, one may manually scan the references for additional articles to review. Manual scanning is a laborious task that only yields limited positive results. This method, while still laborious, expands the iterative flow as it identifies additional research articles. How could this process of Backward Citation Expansion be accelerated and made to provide more value? A more effective technique for providing targeted research was developed, the Feedback Research Process shown in Exhibit 4.

The Feedback Research Process begins with Backward Citation Expansion, feeds the selected articles into the data set of information, and determines logical candidates for additional research. The process determines suggested articles, suggested authors, and suggested journals, enabling the researcher to refine their research more quickly. The Feedback Research Process places citations in context of the topic being researched, which is a key benefit over citation expansion. The Feedback Research Process works by parsing the entire reference section of a selected article. Placing the references in a database provides a collection and analysis location for the information set. Joining new articles with other selected articles in the data set allows the analysis of the commonalities between the articles.

Analyzing the data is performed with each article added to the data set. The first step is to curate the reference data added. Parsing PDF files for reference sections is many times a waste of effort. Many PDF files are imaged so that there is no text to search. Many have no OCR data track. Using databases such as Cross-Ref is a reasonable method to help clean up some data. Many times, the data needs to be cleaned up by hand.

With all references for all articles added to the database and entries curated, identifying the articles of interest becomes relatively simple. Articles of interest are revealed from the data set when the database is sorted by the article title. Identify all articles not already in the data set with multiple reference listings. The more times the reference is listed, the more likely it will be directly applicable to the research topic. Finding Authors of interest can be obtained through a similar procedure. Sort the data set by the author. The more times the author appears in the data set, the more prominent the author is for the topic of research. Knowing key or prominent authors allows one to investigate or search more directly for additional articles and information focused on the research topic. Sorting the data set by journal works similar to authors of interest. The more times a particular journal appears, the more prominent the journal is in the given research topic. Again, this permits the researcher to focus directly on journals that will be more likely to contain articles on the given topic. The Feedback Research Process takes backward citation expansion to the next level finding articles faster and more accurately.

Conclusions

Whether in patent citation research or general research, the connectivity of articles to references or cited articles to their citations creates a directed network graph that can be utilized to visualize the results (Park, Lee, Kim, & Lee, 2018). The directed graph visually shows the breadth of research conducted for an article. However, the citation expansion methods described and their visualization do not focus the research on the central articles for the specific research topic being studied. Using Backward Citation Expansion, Forward Citation Expansion, Feedback Research process, or another method, the goal is the same. Literature research aims to find the appropriate material for the subject under consideration regardless of the methodology used.

In neither Backward nor Forward Citation Expansion, the count of citations was a point of discussion. Choosing citations based on citation count may be enticing. Receiving citations for one’s article is also preliminarily invigorating. However, this is merely the Facebook “Like” Effect (Haruvy & Popkowski Leszczyc, 2018). Was it cited because the article was making a valid conclusion based on scientific evidence, or was the author using it as a contradictory example of some point? From just a citation, one cannot determine the value or lack thereof of the citation. A citation alone, therefore, does not provide value without context. Without context, one could naturally conclude that a citation is nothing more than a Facebook “Like.” Context matters.

With Google Scholar and other databases returning search results in the millions, it is impractical to utilize keyword search as the only mechanism to identify articles. The Feedback Research Process focuses the research by adding context to references. The Feedback Research Process identifies articles of interest more quickly, authors of interest, and journals of interest which is the goal of performing the research in the first place.

References

Briscoe, S., Bethel, A., & Rogers, M. (2020). Conduct and reporting of citation searching in Cochrane systematic reviews: A cross‐sectional study. Research Synthesis Methods, 11(2), 169-180.

Chen, C. (2018). Cascading citation expansion. arXiv preprint arXiv:1806.00089.

Chen, C., Lin, X., & Zhu, W. (2006). Trailblazing through a knowledge space of science: Forward citation expansion in CiteSeer. Proceedings of the American Society for Information Science and Technology, 43(1), 1-17.

Haruvy, E., & Popkowski Leszczyc, P. T. (2018). The influence of social media on charitable fundraising. Available at SSRN 3201494.

Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P., . . . Moher, D. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS medicine, 6(7), e1000100.

Park, Y.-N., Lee, Y.-S., Kim, J.-J., & Lee, T. S. (2018). The structure and knowledge flow of building information modeling based on patent citation network analysis. Automation in Construction, 87, 215-224.

About the Author

Ben Park, Ph.D., completed his doctorate in Industrial Engineering with an emphasis in Management Systems at the University of Central Florida (UCF), Orlando. Dr. Park has a Master's degree in Engineering Management and a BS in Computer Engineering, both from UCF. Dr. Park is a software engineering and development leader with 30 years’ experience developing and deploying custom-built software solutions. As CTG’s Director of Software Development, he leads a team of software development professionals that build flexible solutions to meet the needs of enterprise clients across industries. Dr. Park is a proven, motivated, and enthusiastic leader that understands how to apply a strategic vision to practice, seeks and forms collaborative teams, and transforms groups into teams aligned to a common vision. Dr. Park is an award-winning technical leader with the knowledge to design large systems of systems as well as small, embedded devices. With a Ph.D. focus in globally distributed teams using agile software development, he has a clear understanding of what is needed to operate in multiple time zones, locations, and cultures.

Timothy Kotnour completed his doctorate in Industrial & Systems Engineering with an emphasis in Management Systems Engineering at Virginia Tech in Blacksburg, Virginia. He completed his Bachelor of Science in Industrial Engineering at Bradley University in Peoria, Illinois. He is the Lockheed Martin St. Laurent Professor in the Department of Industrial Engineering and Management Systems at the University of Central Florida. He is the Director of the UCF Engineering Leadership and Innovation Institute and the Program Director of the Professional Engineering Management Program. He is the author of the books “Transforming Organization: Strategies and Methods” and “Inspiring the Leader Engineer: Instilling the Burning Desire and Confidence to Change the World.” Dr. Kotnour has been awarded three NASA Public Service Medals (2016, 2005, and 2001) for the partnership work with the Kennedy Space Center. He is also a Fellow of the American Society for Engineering Management (ASEM). He was past editor of ASEM’s Engineering Management Journal.