Public
Edited
Jul 15, 2021
Importers
1 star
About SOC and O*Net
Standard Occupational Classification (SOC) and the O*Net Online System
There are 35 skills and 997 occupations, organized by the federal Standard Occupational Classificaiton (SOC) System and the O*Net online service into 5 skill sets and 22 occupations. The ontological objective is to organize the skills and occupations along BFO-based ontological lines to see if a more "reason-able" arrangement would be beneficial.
Skills are divided into 5 groups in the O*Net System by horizontal lines:
Basic,
Social,
Technical (which includes Complex Problem Solving),
Systems,
Resource Management.
The proposed ontological grouping of skills is also 5 groups based on a gradient of cognitive challenge for each skill, with the most challenging at the top:
Advanced: most cognitively challenging skills, including Complex Problem Solving and selected skills from the Basic, Technical and Systems groups; 10 total.
RM: challenging Resource Management; 4 total.
Sound: less challenging Basic; 7 total.
System: less challenging Systems; 6 total.
Tech: less challenging technical skills; 8 total with 4 at one level and 4 at the lowest level, for a total of 35.
Showing all occupations is not possible, so they may be filtered in two ways:
modulo-4 249 at a time, selecting either residue 0 or 2 (eg 14 modulo = remainder 3 and residue 2).
by O\*Net Major Group (of which there are 22 non-military-specific groups) within an ontological grouping (3-level hierarchy with two main divisions, four sub-divisions and ten sub-sub-divisions).
Each mark represents a skill-occupation pairing with the following legend:
Importance is shown by shape, upper left triangle for high, lower left triangle for "above" and rectangle for low. Above is between an average importance for that skill and one standard deviation above the average.
Level of expertise is shown by color, red for high, blue for above average, gray for below average metric for that skill level of expertise.
obscfg.widthMax < width ? obscfg.widthMax : width
tmpShift = (200 - sliderPctDisp) - 52
tmpWbyHor = width/obscfg.pxHor
// cell: Drag object of functions to effect the drag operation on slider rece
// and translate display accordingly
// 2021-06-13 See Word doc on doing this Ntbk (2021-06-13) Drag SVG as Group
drag = {
function dragstarted(event, d) {
// upon click to drag, the group moves horizontally to some point before dragging
// what it is doing is positioning the g at the point of the click; can this be overridden and
// only capture x,y when the mouse is dragged?
console.log("drag start: How to avoid any move? " + event.x)
d3.select(this).attr("stroke", "black")
}
function dragged(event, d) {
if (false) console.log("what is in event? ", event.x)
// translate only the x dimension:
console.log("dragged " + event.x + " THIS.... ", d3.select(this))
// d3.select(this).raise().attr("transform", `translate(${event.x})`)
d3.select(this).attr("x", event.x) // did not work right: ("transform", `translate(${event.x})`)
}
function dragended(event, d) {
// get x of mainGroup first determine delta x
const priorX = d3.select("#sliderTst")
console.log("Is there an event here? ", event)
console.log("Here is prior rect obj ", priorX)
console.log("Here is prior rect val " + priorX._groups[0][0].attributes.x.nodeValue)
const deltaX = event.x - priorX.x
console.log("zzzz ", deltaX)
// translate the mainGroup by a delta: slider dx will mean - test for now
// change of x in group = (newSilderX - oldSliderX)
let test = d3.select("#sliderTst")
.attr("transform", `translate(40)`)
.attr("fill", "green")
d3.select(this).attr("stroke", "gray") // reset
}
return d3.drag()
.on("start", dragstarted)
.on("drag", dragged)
.on("end", dragended)
}
function sliderClick(event, d) {
console.log("slider clicked " + event.x)
d3.select("#slider")
.attr("x", 120)
}
obsMessage = "Any old message here!"
obsStatsBySkill = fnCalcStatsBySkill()
obsStatsByOccup = fnCalcStatsByOccup()
// cell: test
fnGetPoints("11-1011.00")
// cell: Function to get all display points for a given occupation id; SKIP SKILL ELEMENT ZERO!
// Reference: obsSkillSeq = O*Net Sequence or else OccO onto, so order skills accordingly
// 2021-06-11 hopefully getting close!
// 2021-06-22 Definitely getting close - implementing onto skill sequence
// 2021-06-25 Revised onto skills considerably and changed CSV headings, a mess!
function fnGetPoints(occupId) {
// myPoints will be the 35 skill indicators + 36th for occup index level,
// each entry will be x, y and path
// skillIndices will be array of skill idx for desired sequence, based on ONet index
// (if sequence is onet it is the dict_Skills index)
let myPoints = [] // new Array[37].fill(0) // symbol info for this occup
const oc = dict_Occup[occupId]
let numLvAbv = 0
let numLvHigh = 0
// stats array is zero based, 0-34 = skills and index 35 is num LvAbv and LvHi
// above does not appear to be correct
// 6/12 it looks like entry zero is not filled so skip it!
// 6/22 retrieve the ith index from the skills index if O*Net but [ontoIdx] if OccO skill sequence
// this is a bit tricky: skill info is in obsArSkillsAndStats; skillSeqIdx = sequenced indexed of sorted abilities
// skillIdx = the skill's original O*Net index 1-35
console.log("what is skills and stats? ") // it is right there!
for (let skillSeqIdx = 1; skillSeqIdx < 36; skillSeqIdx++) { // skillSeqIdx is the display (sorted) sequence
// all needed is shape info. Get im and lv levels, 0, 1, 2 for low, abv, high and set path
// NOTE: if Onto sequence is desired, the processing is done by onto index, so it will not be consecutive order:
const skillIdx = obsSkillSeq === "O*Net Sequence" ? skillSeqIdx : obsArSkillIdxOnto[skillSeqIdx].skillOntoIdx
// console.log("wwwwwahatt? ?? ", obsArSkillIdxOnto[1])
const currSkill = obsArSkillsAndStats[skillIdx]
if (occupId === "11-1011.00") console.log("Onto chf ex: " + skillSeqIdx + " skill Onet idx: " +
obsArSkillIdxOnto[1].skillOntoIdx + " skillIdx: " + skillIdx + " xx ",
obsArSkillsAndStats[skillIdx])
let imVal = 0 // default
if (oc.im[skillIdx] > currSkill.imAvg + currSkill.imStdDev) {
imVal = 2 // imHigh
} else if (oc.im[skillIdx] > currSkill.imAvg) {
imVal = 1 // imAbv
}
let lvVal = 0 // default low
if (oc.lv[skillIdx] > currSkill.lvAvg + currSkill.lvStdDev) {
lvVal = 2 // lvHigh
} else if (oc.lv[skillIdx] > currSkill.lvAvg) {
lvVal = 1 // lvAbv
}
// get numbers for occup index if im > low for the skill
if (oc.im[skillIdx] > currSkill.imAvg) {
if (oc.lv[skillIdx] > currSkill.lvAvg + currSkill.lvStdDev) {
numLvHigh += 1
/* console.log("High " + oc.lv[skillIdx] + " avg " +
currSkill.lvAvg + " stdd " +
currSkill.lvStdDev) */
} else if (oc.lv[skillIdx] > currSkill.lvAvg) {
numLvAbv += 1
}
}
// we have results for this Occup-skill combination - add to myPoints
myPoints.push({ imVal: imVal, lvVal: lvVal })
}
// we have results for this occupation; push the two occupation index metrics
// onto the array as element 36 with numLvAbv, numLvhigh in the two spots
myPoints.push({ imVal: numLvAbv, lvVal: numLvHigh })
// order skill metric values by seqSkill
// console.log("xxxx " + numLvAbv + " ", myPoints)
return myPoints
}
fnCalcEntMajGrpSkills("ONet")
fnCalcEntMajGrpSkills("Random")
/* 7/2/21 5:50pm - worn out! got the "aggregate" sum of entropies for each group - skill
entropy for each skill - group are calculated by fnCalcEntMajGrpSkills(ONet or Random)
and results (sum only not recomputed running variance) is computed in obsResultsAggregate
To do: process by group ? << think this through - if skills are aggregated, is that a good idea? I think so
BUT: Do we get a running variance across skills? I think not! Best just to use a SUM!!
Then once we have a sum per group, get the running variance for the groups
Quite a number of cells need to be redone / deleted:
*/
obsResults1 = {
const resONetIm = fnCalcEntMajGrpSkills("ONet")
const resRandIm = fnCalcEntMajGrpSkills("Random")
// Phase I: grand total entropy, all groups
let sumForSkill5ONetIm = 0
let sumForSkill5RandIm = 0
let sumForSkillsONetIm = 0
let sumForSkillsRandIm = 0
for (let grpnum = 0; grpnum < resONetIm.length; grpnum++) {
for (let skillnum = 1; skillnum < 36; skillnum++) {
if (skillnum === 5) {
sumForSkill5ONetIm += resONetIm[grpnum][skillnum]
sumForSkill5RandIm += resRandIm[grpnum][skillnum]
}
sumForSkillsONetIm += resONetIm[grpnum][skillnum]
sumForSkillsRandIm += resRandIm[grpnum][skillnum]
}
}
// Phase IIa: Entropy for each Group, all skills
// NOW: can the ONet results for each group be combined to yield an overall all-group entropy?
// (this does not apply to the random case; however, since they are grouped similarly, calculate this as well)
// process: get the running variance across groups
// step one is get group ent across all skills:
let sumForGrpsONetIm = []
let sumForGrpsRandIm = []
let priorONetIm = 0
let priorRandIm = 0
for (let grpnum = 0; grpnum < resONetIm.length; grpnum++) {
for (let skillnum = 0; skillnum < 36; skillnum++) {
}
}
/* for (let grpnum = 0; grpnum < resONetIm.length; grpnum++) {
if (grpnum > 0) {
sumForGrpsONetIm += Math.pow(resONetIm[grpnum] - priorONetIm, 2)
console.log("RExx1111xxx " + resONetIm[grpnum] + " val " + priorONetIm)
sumForGrpsRandIm += Math.pow(resRandIm[grpnum] - priorRandIm, 2)
priorONetIm = resONetIm[grpnum]
priorRandIm = resRandIm[grpnum]
console.log("RExxxxx " + sumForGrpsONetIm + " val " + resONetIm[grpnum])
}
} */
// Pase IIb: combine group entropies - get sum of squares then sqrt of avg
// get average using 21 group differences in the 22 groups
sumForGrpsONetIm = Math.sqrt(sumForGrpsONetIm / 21)
sumForGrpsRandIm = Math.sqrt(sumForGrpsRandIm / 21)
// Phase I results:
// return { ONet: Math.round(sumForSkillsONetIm * 100)/100, Rand: Math.round(sumForSkillsRandIm * 100)/100 }
// Phase II results:
return { ONet: Math.round(sumForGrpsONetIm * 100)/100, Rand: Math.round(sumForGrpsRandIm * 100)/100 }
}
obsResultsAggregate = {
const resONet = fnCalcEntMajGrpSkills("ONet")
const resRand = fnCalcEntMajGrpSkills("Random")
let sumForSkill5ONetIm = 0
let sumForSkill5RandIm = 0
let sumForSkillsONetIm = 0
let sumForSkillsRandIm = 0
for (let grpnum = 0; grpnum < resONet.length; grpnum++) {
for (let skillnum = 1; skillnum < 36; skillnum++) {
if (skillnum === 5) {
sumForSkill5ONetIm += resONet[grpnum][skillnum]
sumForSkill5RandIm += resRand[grpnum][skillnum]
}
sumForSkillsONetIm += resONet[grpnum][skillnum]
sumForSkillsRandIm += resRand[grpnum][skillnum]
}
}
return { ONet: Math.round(sumForSkillsONetIm * 100)/100, Rand: Math.round(sumForSkillsRandIm * 100)/100 }
}
// ARCHIVAL: function to get skill entropies for each group*****
// cell: For each of the 22 ONet major-group-index-sets of desired sequence calculate LeafOntoEntropy
// Param: seq is either ONet or Random
// issue 1: dict_Occup is accessed by OccupId not index
// issue 2: the seqRandom is a random number - needs to be sorted to get an index
// 7/2/21 reference is now a dictionary by OccupId with occupid's for both ONet and Random
function fnCalcEntMajSkillGrps(seq) {
console.log("Hixxxx zzz")
let grpEnt = 0 // for now make it the sum of skill variances Array(36).fill(0)
let allGrpEnt = [] // array for all groups
let numGrps = 0
// const occupLookup = Object.entries(dict_Occup) // will have array of [key, value]
let arSumSquaresIm = Array(36).fill(0) // sum of sqares of adjecent occups for each of 35 skills skipping zero
let arPriorIm = Array(36).fill(0) // im value of prior occup for each skill
// Process each group, and within each group, process all occupations
// process each group:
for (const [key, val] of Object.entries(obsEntMajGrpOccupIds)) {
if (numGrps < 25) {
console.log("Doing group " + numGrps)
let numOccup = 0
// process each occup:
val.forEach((ref, j) => {
const occupId = seq === "ONet" ? ref.ONetOccupId : ref.RandomOccupId
const myOc = dict_Occup[occupId]
// ascertain im has nonzero values
let tot = myOc.im.reduce((a, b) => a + b, 0)
if (tot > 0) {
// get occup from dict_Occup
if (numOccup < 4 && j < 3) console.log("doing occup " + occupId + " keyoc ", myOc.im)
if (numOccup > 0) {
// calculate "running variance" - degree of change from adjacent (leaf) value
for (let i = 1; i < 36; i++) { //*** make 36
arSumSquaresIm[i] += Math.pow(myOc.im[i] - arPriorIm[i], 2)
// console.log("im val= " + myOc.im[i] + " ri " + arPriorIm[i] + " Sq " + arSumSquaresIm[1])
}
arPriorIm = [...myOc.im] // copies actual values; can we just use ref to prior occup?
}
}
numOccup += 1
})
// get sqrt of average
for (let i = 1; i < 36; i++) arSumSquaresIm[i] = Math.sqrt(arSumSquaresIm[i] / numOccup)
numGrps += 1
allGrpEnt.push(arSumSquaresIm)
}
}
console.log("result im: ", arSumSquaresIm)
return allGrpEnt
}
// *** REDO / DELETE this just get sum:
// cell: function to process output of fnCalcEntMajGrpSkills (whether ONet or Random) and return
// input: array of groups with each group having 36 skill entropy values
// Returns: array of occup groups with a total entity for all skills
function fnCalcEntMajGrps(entMajGrpSkills) {
// First step is get aggregate entropy across all skills
let entForMajGrps = [] // results
let priorSkillEnt = 0
const numGrps = entMajGrpSkills.length
for (let grpnum = 0; grpnum < numGrps; grpnum++) {
if (grpnum > 0) {
for (let skillnum = 0; skillnum < 36; skillnum++) {
// sum of squares diff from prior group
entForMajGrps[grpnum] += Math.pow(entMajGrpSkills[skillnum] - priorSkillEnt, 2)
}
priorSkillEnt = entMajGrpSkills[skillnum]
}
// get sqrt of avg of 22 groups considering there are 21 differences (resOnet
entForMajGrps[grpnum] = Math.pow(entForMajGrps[grpnum] / (numGrps - 1))
}
return entForMajGrps
}
// *** this is a newer version I think? THE ONE USED is fnCalcEntMajSkillGrps << MIS-NAMED!
// cell: For each of the 22 ONet major-group-index-sets of desired sequence calculate LeafOntoEntropy
// which of these is used?
// Param: seq is either ONet or Random
// returns array of 22 groups each having ent figure for array of 25 skills
// issue 1: dict_Occup is accessed by OccupId not index
// issue 2: the seqRandom is a random number - needs to be sorted to get an index
// 7/2/21 reference is now a dictionary by OccupId with occupid's for both ONet and Random
function fnCalcEntMajGrpSkills(seq) {
console.log("Hixxxx zzz")
let grpEnt = 0 // for now make it the sum of skill variances Array(36).fill(0)
let allGrpEnt = [] // array for all groups
let numGrps = 0
// const occupLookup = Object.entries(dict_Occup) // will have array of [key, value]
let arSumSquaresIm = Array(36).fill(0) // sum of sqares of adjecent occups for each of 35 skills skipping zero
let arPriorIm = Array(36).fill(0) // im value of prior occup for each skill
// Process each group, and within each group, process all occupations
// process each group:
for (const [key, val] of Object.entries(obsEntMajGrpOccupIds)) {
if (numGrps < 25) {
console.log("Doing group " + numGrps)
let numOccup = 0
// process each occup:
val.forEach((ref, j) => {
const occupId = seq === "ONet" ? ref.ONetOccupId : ref.RandomOccupId
const myOc = dict_Occup[occupId]
// ascertain im has nonzero values
let tot = myOc.im.reduce((a, b) => a + b, 0)
if (tot > 0) {
// get occup from dict_Occup
if (numOccup < 4 && j < 3) console.log("doing occup " + occupId + " keyoc ", myOc.im)
if (numOccup > 0) {
// calculate "running variance" - degree of change from adjacent (leaf) value
for (let i = 1; i < 36; i++) { //*** make 36
arSumSquaresIm[i] += Math.pow(myOc.im[i] - arPriorIm[i], 2)
// console.log("im val= " + myOc.im[i] + " ri " + arPriorIm[i] + " Sq " + arSumSquaresIm[1])
}
arPriorIm = [...myOc.im] // copies actual values; can we just use ref to prior occup?
}
}
numOccup += 1
})
// get sqrt of average
for (let i = 1; i < 36; i++) arSumSquaresIm[i] = Math.sqrt(arSumSquaresIm[i] / numOccup)
numGrps += 1
allGrpEnt.push(arSumSquaresIm)
}
}
console.log("result im: ", arSumSquaresIm)
return allGrpEnt
}
d3 = require("d3@6")
Observable Notebook Self-documentation
Main Display Cells
mainPageNested - Main display of matrix;
Input Controls - Affecting Output Display
obsSelType = group of occupations to process: Ref-200, Modulo-4 (0 or 2 for first, third), others (later)
This controls the set of occupations processed by setting xxxxxyyyyzzz
obsOccupSeq = occupation sequence: Major O*Net Group, OccO Onto Group, or Random
This controls the sequence of occupations displayed horizontally
Upon change the occupations are sorted by Onto Group (***FIELD??) and vertical line dividers between occup groups are set in obArOccupGrpIndices
obsSkillSeq = O*Net sequence or Skill Onto Group
This controls the vertical ordering of skills, with indices between skill groups in divHorSkillsONet and divHorSkillsOccO
Input Data, and Related Array or Dictionary
data_Occup; converted to dict_Occup
Note: This also contains the 36-entry arrays for 36 im and lv values (entry zero unused)
filename = O-Net B_Occupations SOC-Related (2021-06-10).csv
data_SkillOccup; remains as read from .json file, array of objects
filename = O-Net C_Skill-Occupation with Metrics (2021-06-10).csv
data_Skills; converted to dict_Skills << Note: dict_Skills appears NOT to be used!
filename = O-Net D_Skills_TreeOnto (2021-06-10).csv
Derived Data: Processing Objects derived from Input
obsArSelOccupIds: the MAIN ordered list of occupation ids in the desired sequence (ONet or Onto)
obsArSkillsAndStats: Main reference for display of skills: skillId (from Element_ID in SkillOccup), ONetIdx (as-is), OntoIdx: Onto Seq and the average and standard distribution for each skill across all occupations.
obsArOccupGrpIndices: currently dispayed occupation indices at which groups change for vertical dividers
obsArSkillIdxOnto: array for each skill in ONTOLOGICAL order showing the ONetIdx (original, as-read) index for accessing skills table
Output Entities that appear in Main Page
obsMessage -- attempt to in effect put console log output on display; did this work out?
Functions
fnSkillsOccupAnalyze
fnOccupAnalyze
drag {} - object that contains several functions: dragstarted, draffed, dragended to handle the drag event
use or get rid of: dictOccupSkills, obsStatsBySkill, fuCalcStatsBySkill, obsStatsByOccup, finCalcStatsByOccup
- *** OR: usee calc stats to do entropy!!
- fnCalcStatsBy
### Imports
- Tooltip from "@clhenrick/tooltip-component"
### Standard Notebook cells
- d3
- obscfg - configuration info (window size, margins, etc) plus app-specific constants:
- Skill level colors
- divHorSkillsONet - the indices at which skill group horizontal line appears in O*Net sequence change
- divHorSkillsOccO - same, when in OccO Onto skill sequence
Other?
### Coding Standards
- obscfg execution options for margin, height, width if not full
- obs prefix for an app variable – assumed to be an object not array
- obsAr Prefix for an app variable that is an array of values or objects
- data_ Prefix for data read in from JSON or CSV as array of objects
- dict_ Prefix for input data converted to a dictionary with key, value
- fn Prefix for a cell that is a function
- obsFn Prefix for a cell that is an object containing functions
- Example: drag has dragstarted, dragged, and dragend functions
- Avoid variable names that are identical to common library variables (eg. x, y, value, key)
Purpose-built for displays of data
Observable is your go-to platform for exploring data and creating expressive data visualizations. Use reactive JavaScript notebooks for prototyping and a collaborative canvas for visual data exploration and dashboard creation.
